Child pages
  • Terracotta Operations Center

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin
Include Page
documentation-navbar
documentation-navbar
Section
Column
width250px
{div:style=} {include:
Wiki Markup
Div
stylemargin-top:20px
Include Page
documentation-index
documentation-index
} {div}
Column
{div9:id=documentationcolumn} {include:HeaderBasic} h1. Terracotta Operations Center {toc:minLevel=2|maxLevel=3} h2. Introduction The Terracotta Operations Center delivers a full-featured monitoring and diagnostics tool for operators of an application clustered with Terracotta. The Operations Center is available with Enterprise versions of Terracotta. {tip} The Operations Center is designed to monitor and manage a Terracotta cluster in a production environment. In a development environment, use the [Terracotta Developer Console]. {tip} The Operations Center is a GUI console that includes a JMX client. Using the console, you can: * Monitor the health of servers and clients under changing conditions. * Back up the shared-object database * Force distributed garbage collection operations * View cluster-wide statistics * Disconnect problem clients remotely * Shutdown servers remotely * Receive console status and server log messages in a static window These and other console features are described below. {tip} Many of the images in this document can be viewed in a larger format by clicking them. {tip} h2. Launching the Terracotta Operations Center You can launch the Terracotta Operations Center from a command line. h5. Microsoft Windows {{\[PROMPT\]
Wiki Markup
Div
iddocumentationcolumn
Include Page
HeaderBasic
HeaderBasic

Terracotta Operations Center

Table of Contents
minLevel2
maxLevel3

Introduction

The Terracotta Operations Center delivers a full-featured monitoring and diagnostics tool for operators of an application clustered with Terracotta. The Operations Center is available with Enterprise versions of Terracotta.

Tip

The Operations Center is designed to monitor and manage a Terracotta cluster in a production environment. In a development environment, use the Terracotta Developer Console.

The Operations Center is a GUI console that includes a JMX client. Using the console, you can:

  • Monitor the health of servers and clients under changing conditions.
  • Back up the shared-object database
  • Force distributed garbage collection operations
  • View cluster-wide statistics
  • Disconnect problem clients remotely
  • Shutdown servers remotely
  • Receive console status and server log messages in a static window

These and other console features are described below.

Tip

Many of the images in this document can be viewed in a larger format by clicking them.

Launching the Terracotta Operations Center

You can launch the Terracotta Operations Center from a command line.

Microsoft Windows

[PROMPT] %TERRACOTTA_HOME%\bin\ops-center.bat

}} h5.

UNIX/Linux
{{\

[PROMPT

\

]

$

\

{TERRACOTTA_HOME

\

}/bin/ops-center.sh&

}} h3. The Console Interface When the console is not connected to a server, the selection buttons in the control panel on the left are inactive. !tocDisconnected.png|thumbnail, vspace=8! The connection panel contains the last host and JMX port entered. The first time the console is started, default values

The Console Interface

When the console is not connected to a server, the selection buttons in the control panel on the left are inactive.

Image Added

The connection panel contains the last host and JMX port entered. The first time the console is started, default values (host=localhost,

jmx-port=9520)

are

used.

{

Info
:
title
=
Setting
a
JMX
Port
}

To

learn

more

about

setting

JMX

ports,

see

the

[

Configuration

Guide

and

Reference|Configuration Guide and Reference#jmx-port]. {info} Once the console is connected to a cluster, the selection buttons in the control panel become active. The console can connect to only one cluster at a time. The status bar at the bottom of the console window displays messages on the latest changes in the cluster, such as nodes joining or leaving. h3. Context-Sensitive Help Context-sensitive help is available wherever !Admin Console Guide^HelpButton.png! (help button) appears in the Terracotta Operations Center. Click !Admin Console Guide^HelpButton.png! in a console panel to open a web-browser page containing help on the features in that panel. h3. Context Menus Some console features have a context menu that is accessed by right-clicking the feature. For example, to open a text-management context menu in the Logging panel, right-click in the panel. h2. Connecting and Disconnecting From a Cluster To connect to an existing cluster, enter the hostname and JMX port number of a Terracotta server in the cluster, then click the *Connect* in the connection panel. To automatically connect to a cluster whenever the console starts or when at least one of the cluster's servers is running, enable *Connect automatically* in the cluster context menu. Automatic connections are attempted in the background and do not interfere with normal console operation. h3. Connecting to a Secured Cluster A Terracotta cluster can be secured for JMX access, requiring authentication before access is granted. Connecting to a secured cluster prompts users to enter a username and password. For instructions on how to secure your Terracotta cluster for JMX, see the [Configuration Guide and Reference|Configuration Guide and Reference#server-authentication]. h3. Disconnecting from a Cluster To disconnect from a cluster, click *Disconnect* in the control panel. h2. Dashboard The Dashboard provides live measurements of the most important metrics affecting the cluster. This panel is where you monitor the components and health of your cluster. !tocDashboard.png|thumbnail, vspace=8! To open the Dashboard, click *Dashboard* in the control panel. h3. Graphs Runtime statistics graphs provide both live measurements and historic trends of resource usage and the work performed by the Terracotta cluster. !Terracotta Developer Console^OptionsDialog.png|align=right! {anchor:OptionsDialog} To adjust the graphs' poll and history periods, click *Options* on the Control panel. In the *Options* dialog, adjust the values in the polling and history fields. These values apply to all runtime-statistics views. Specific runtime statistics are defined in the following sections. {anchor:ServerCpuUsage} h4. Host CPU Usage Shows the CPU load as a percentage. If more than one CPU is being used, each CPU's load is shown as a separate graph line. {anchor:ServerHeapUsage} h4. Heap Usage Shows the amount, in megabytes, of maximum available heap and heap being used. {anchor:TransactionRate} h4. Write Transaction Rate Shows the number of completed Terracotta transactions. [Terracotta transactions|Concept and Architecture Guide#Transactions] are sets of one or more clustered object changes, or writes, that must be applied atomically. h3. Unacknowledged Client Transactions Every [Terracotta transaction|Concept and Architecture Guide#Transactions] in a Terracotta cluster must be acknowledged by Terracotta clients with in-memory shared objects that are affected by that transaction. For each client, Terracotta server instances keep a count of transactions that have not been acknowledged by that client. The Unacknowledged Transaction Broadcasts statistic is a count of how many transactions the client has yet to acknowledge. The Unacknowledged Client Transactions bar graph shows the number of unacknowledged Terracotta transaction broadcasts for each client in the cluster. !Admin Console Guide^ServerPendingTxns.png|thumbnail, vspace=8! This graph is useful for quickly locating and investigating clients that maintain a large or growing number of unacknowledged Terracotta transaction broadcasts. In a healthy cluster, clients clear unacknowledged transactions as they process transactions. The bar representing each client's unacknowledged-transactions volume fluctuates as each client processes its acknowledgements. However, clients experiencing trouble may not be able to clear unacknowledged transactions, and the bars corresponding to those clients remain in place or even grow. Such clients could slow down an entire cluster and may need to be disconnected. h3. Activity Gauges {include:Console Dashboard Include} h2. Manage The Manage panel displays a table of all connected servers and clients in the cluster. This panel is where you can check the status of servers and clients, remove them from the cluster, and get more detailed information. !tocManage.png|thumbnail, vspace=8! To open the Manage panel, click *Manage* in the control panel. {anchor:Servers} h3. Working with Servers The Manage panel lists each server by name and gives its current status. There must be at least one active server instance for the cluster to function. In a Terracotta server array, a typical mirror group has one active server instance and one passive (or "hot" standby) backup. A server's role within the cluster is shown in the center column. {anchor:server status} A status light next to each server's name indicates the server's _connection_ status. The light's color indicates the server's current connection status. A cluster can have one server, or be configured with multiple servers that communicate state over the network or use a shared file-system. The following table summarizes the connection status lights. || Status Light || Server Status || Notes || | *{color:green}GREEN{color}* | Active | The server is connected and ready for work. | | *{color:red}RED{color}* | Unreachable | The server, or the network connection to the server, is down. | | *{color:yellow}YELLOW{color}* | Starting or Standby | A server is starting up; in a disk-based multi-server cluster, a passive server goes into standby mode until a file lock held by the active server is released. | | *{color:orange}ORANGE{color}* | Initializing | In a network-based multi-server cluster, a passive server must initialize its state before going into standby mode. | | *{color:cyan}CYAN{color}* | Standby | In a network-based multi-server cluster, a passive server is ready to become active if the active server fails. | {anchor:server panel} h4. Server Panel Click *Details >>* under a specific server to display tabbed subpanels with detailed information about that server. Click *<< Back* to return to the main Manage panel. The *Main* tab displays the server status and a list of properties, including the server's IP address, version, license, and persistence and failover modes. The *Environment* tab displays the server's JVM system properties and provides a *Find* tool. The *Config* tab displays the Terracotta configuration the server is using and provides a *Find* tool. h4. Connecting and Disconnecting from a Server The console is connected to all of a cluster's servers when it's connected to the cluster. Being connected to a server means that the console is listening for JMX events coming from that server. {note} If you have confirmed that a Terracotta server is running, but the console is unable to connect to it, a firewall on your network may be blocking the server's JMX port. {note} The console is disconnected from a cluster's servers when it's disconnected from the cluster. The console is also disconnected from a server when that server is shut down, but the server still appears as part of the cluster. However, its [connection status|#server status] changes. To shut down a server, click the *Shutdown* button in that server's row. A dialog prompts to you confirm the shutdown. {anchor:Clients} h3. Working with Clients The Manage panel lists each client by name and gives its current status. {anchor:clientpanel} h4. Client Panel Click *Details >>* under a specific client to display tabbed subpanels with detailed information about that client. Click *<< Back* to return to the main Manage panel. The *Main* tab displays a list of client properties such as hostname and DSO port. The *Environment* tab displays the client's JVM system properties and provides a *Find* tool. The *Config* tab displays the Terracotta configuration the client is using and provides a *Find* tool. h4. Connecting and Disconnecting Clients When started up properly, a Terracotta client is automatically added to the appropriate cluster. When a Terracotta client is shut down or disconnects from a server, that client is automatically removed from the cluster and no longer appears in the console. {anchor:ClientDisconnect} To disconnect a client from the cluster, click the *Disconnect* button in that client's row. A dialog prompts to you confirm the shutdown {anchor:backup} h2. Backup Using the Terracotta backup feature, you can create a backup of the data being shared by your application. The backup feature is available from the Backup panel. To open the Backup panel, click *Backup* in the control panel. If the cluster's persistence mode is not set to {{permanent-store}}, then the backup feature is disabled (see [#Enabling Backups]. !tocBackupDisabled.png|thumbnail, vspace=8! If backups are enabled, click *Backup DB* to perform a backup. A dialog box appears where you can confirm the backup destination directory or enter a new destination. The backup is saved to the directory {{objectdb}} at the destination. The database is always backed up to a directory called {{objectdb}}, which is automatically created if it does not exist at the destination. {tip} Backups can also be initiated using the backup-data script (see _Tools Catalog_ in {html}

Reference.

Once the console is connected to a cluster, the selection buttons in the control panel become active. The console can connect to only one cluster at a time.

The status bar at the bottom of the console window displays messages on the latest changes in the cluster, such as nodes joining or leaving.

Context-Sensitive Help

Context-sensitive help is available wherever Image Added (help button) appears in the Terracotta Operations Center. Click Image Added in a console panel to open a web-browser page containing help on the features in that panel.

Context Menus

Some console features have a context menu that is accessed by right-clicking the feature. For example, to open a text-management context menu in the Logging panel, right-click in the panel.

Connecting and Disconnecting From a Cluster

To connect to an existing cluster, enter the hostname and JMX port number of a Terracotta server in the cluster, then click the Connect in the connection panel.

To automatically connect to a cluster whenever the console starts or when at least one of the cluster's servers is running, enable Connect automatically in the cluster context menu. Automatic connections are attempted in the background and do not interfere with normal console operation.

Connecting to a Secured Cluster

A Terracotta cluster can be secured for JMX access, requiring authentication before access is granted. Connecting to a secured cluster prompts users to enter a username and password.

For instructions on how to secure your Terracotta cluster for JMX, see the Configuration Guide and Reference.

Disconnecting from a Cluster

To disconnect from a cluster, click Disconnect in the control panel.

Dashboard

The Dashboard provides live measurements of the most important metrics affecting the cluster. This panel is where you monitor the components and health of your cluster.

Image Added

To open the Dashboard, click Dashboard in the control panel.

Graphs

Runtime statistics graphs provide both live measurements and historic trends of resource usage and the work performed by the Terracotta cluster.

Image Added

Anchor
OptionsDialog
OptionsDialog

To adjust the graphs' poll and history periods, click Options on the Control panel. In the Options dialog, adjust the values in the polling and history fields. These values apply to all runtime-statistics views.

Specific runtime statistics are defined in the following sections.

Anchor
ServerCpuUsage
ServerCpuUsage

Host CPU Usage

Shows the CPU load as a percentage. If more than one CPU is being used, each CPU's load is shown as a separate graph line.

Anchor
ServerHeapUsage
ServerHeapUsage

Heap Usage

Shows the amount, in megabytes, of maximum available heap and heap being used.

Anchor
TransactionRate
TransactionRate

Write Transaction Rate

Shows the number of completed Terracotta transactions. Terracotta transactions are sets of one or more clustered object changes, or writes, that must be applied atomically.

Unacknowledged Client Transactions

Every Terracotta transaction in a Terracotta cluster must be acknowledged by Terracotta clients with in-memory shared objects that are affected by that transaction. For each client, Terracotta server instances keep a count of transactions that have not been acknowledged by that client. The Unacknowledged Transaction Broadcasts statistic is a count of how many transactions the client has yet to acknowledge.

The Unacknowledged Client Transactions bar graph shows the number of unacknowledged Terracotta transaction broadcasts for each client in the cluster.

Image Added

This graph is useful for quickly locating and investigating clients that maintain a large or growing number of unacknowledged Terracotta transaction broadcasts. In a healthy cluster, clients clear unacknowledged transactions as they process transactions. The bar representing each client's unacknowledged-transactions volume fluctuates as each client processes its acknowledgements. However, clients experiencing trouble may not be able to clear unacknowledged transactions, and the bars corresponding to those clients remain in place or even grow. Such clients could slow down an entire cluster and may need to be disconnected.

Activity Gauges

Include Page
Console Dashboard Include
Console Dashboard Include

Manage

The Manage panel displays a table of all connected servers and clients in the cluster. This panel is where you can check the status of servers and clients, remove them from the cluster, and get more detailed information.

Image Added

To open the Manage panel, click Manage in the control panel.

Anchor
Servers
Servers

Working with Servers

The Manage panel lists each server by name and gives its current status. There must be at least one active server instance for the cluster to function. In a Terracotta server array, a typical mirror group has one active server instance and one passive (or "hot" standby) backup. A server's role within the cluster is shown in the center column.

Anchor
server status
server status

A status light next to each server's name indicates the server's connection status. The light's color indicates the server's current connection status. A cluster can have one server, or be configured with multiple servers that communicate state over the network or use a shared file-system.

The following table summarizes the connection status lights.

Status Light

Server Status

Notes

GREEN

Active

The server is connected and ready for work.

RED

Unreachable

The server, or the network connection to the server, is down.

YELLOW

Starting or Standby

A server is starting up; in a disk-based multi-server cluster, a passive server goes into standby mode until a file lock held by the active server is released.

ORANGE

Initializing

In a network-based multi-server cluster, a passive server must initialize its state before going into standby mode.

CYAN

Standby

In a network-based multi-server cluster, a passive server is ready to become active if the active server fails.

Anchor
server panel
server panel

Server Panel

Click Details >> under a specific server to display tabbed subpanels with detailed information about that server. Click << Back to return to the main Manage panel.

The Main tab displays the server status and a list of properties, including the server's IP address, version, license, and persistence and failover modes.

The Environment tab displays the server's JVM system properties and provides a Find tool.

The Config tab displays the Terracotta configuration the server is using and provides a Find tool.

Connecting and Disconnecting from a Server

The console is connected to all of a cluster's servers when it's connected to the cluster. Being connected to a server means that the console is listening for JMX events coming from that server.

Note

If you have confirmed that a Terracotta server is running, but the console is unable to connect to it, a firewall on your network may be blocking the server's JMX port.

The console is disconnected from a cluster's servers when it's disconnected from the cluster. The console is also disconnected from a server when that server is shut down, but the server still appears as part of the cluster. However, its connection status changes.

To shut down a server, click the Shutdown button in that server's row. A dialog prompts to you confirm the shutdown.

Anchor
Clients
Clients

Working with Clients

The Manage panel lists each client by name and gives its current status.

Anchor
clientpanel
clientpanel

Client Panel

Click Details >> under a specific client to display tabbed subpanels with detailed information about that client. Click << Back to return to the main Manage panel.

The Main tab displays a list of client properties such as hostname and DSO port.

The Environment tab displays the client's JVM system properties and provides a Find tool.

The Config tab displays the Terracotta configuration the client is using and provides a Find tool.

Connecting and Disconnecting Clients

When started up properly, a Terracotta client is automatically added to the appropriate cluster.

When a Terracotta client is shut down or disconnects from a server, that client is automatically removed from the cluster and no longer appears in the console.

Anchor
ClientDisconnect
ClientDisconnect

To disconnect a client from the cluster, click the Disconnect button in that client's row. A dialog prompts to you confirm the shutdown

Anchor
backup
backup

Backup

Using the Terracotta backup feature, you can create a backup of the data being shared by your application. The backup feature is available from the Backup panel. To open the Backup panel, click Backup in the control panel. If the cluster's persistence mode is not set to permanent-store, then the backup feature is disabled (see #Enabling Backups.

Image Added

If backups are enabled, click Backup DB to perform a backup. A dialog box appears where you can confirm the backup destination directory or enter a new destination. The backup is saved to the directory objectdb at the destination. The database is always backed up to a directory called objectdb, which is automatically created if it does not exist at the destination.

Tip

Backups can also be initiated using the backup-data script (see Tools Catalog in

HTML
<a href="/documentation/ga/product-documentation">Terracotta Product Documentation</a>

{html}

).

The

script

is

useful

for

automating

scheduled

backups.

{tip} To change the default backup directory path, edit the

To change the default backup directory path, edit the <data-backup>

property

in

the

Terracotta

server's

configuration

file

with

the

path

to

your

preferred

backup

directory:

<server>

<server>
<data-backup>path/to/my/backup/directory</data-backup>


</server>

h3.

Enabling

Backups

If

a

Terracotta

server

is

not

configured

for

permanent-store

persistence,

the

*

Backup

DB

*

button

is

disabled

and

the

following

message

appears

on

the

*

Backup

database

*

panel:

{

Code Block
}

Backup feature is currently disabled because the cluster is 
operating in temporary-swap-only persistence mode.
{code}

To

enable

the

*

Backup

DB

*

button,

change

the

value

of

the

[

persistence

mode

property

|Configuration Guide and Reference#persistence]

in

the

Terracotta

server's

configuration

file

to

{{

permanent-store

}}

:

{

Code Block
}

<server>
   <dso>
      <persistence>
         <mode>permanent-store</mode>
      </persistence>
   </dso>
</server>
{code} {anchor:restore} h3. Restoring a Backup Terracotta maintains a copy of shared in-memory data on disk. In most server-failure cases, Terracotta automatically restores that shared data by loading it from the copy, recreating the application state. However, if you encounter a situation in which the data files are missing, you can restore them from backups. To restore data files from a backup: # Shut down the Terracotta cluster. # Make copy of any existing data files. # Delete the existing data files. # Copy the backup data files to the directory from which you deleted the original (existing) data files. # Restart the Terracotta cluster. h2. Diagnostics The Diagnostics panels offers tools for monitoring and analysis. There are three tabbed panels available: one for monitoring and triggering the Terracotta Distributed Garbage Collector (DGC), one for viewing cluster events, and one for taking thread dumps. !tocDiagnosticsDGC.png|thumbnail, vspace=8! {tip:title=Operator Events} The cluster events are shown under the Operator Events panel. The button for accessing this panel, which is not shown in the screenshot above, is labeled *Operator Events* and appears between the *Distributed GC* and *Thread Dump* buttons. {tip} To open the Diagnostics panels, click *Diagnositics* in the control panel. {anchor:DistributedGarbageCollection} h3. Distributed Garbage Collection Objects in a DSO root object graph may become unreferenced and no longer exist in the Terracotta client's heap. These objects are eventually marked as garbage in a Terracotta server instance's heap and from persistent storage by the DGC. The DGC is unrelated to the Java garbage collector. {info} For more information on the DGC, see the [Terracotta Concept and Architecture Guide|Concept and Architecture Guide#dgc]. {info} To view a history table of DGC activity in the current cluster, click *Distributed garbage collection* in the cluster list. The history table is automatically refreshed each time a collection occurs. Each row in the history table represents one distributed garbage collection cycle, with the following columns: ||Column ||Definition ||Values |Iteration |The index number of the DGC cycle| Sequential integer| |Type |The type of cycle |Full -- Running a full collection cycle targeting all eligible objects. Young -- Running a collection cycle targeting Young Generation objects. | |Status |The collection cycle's current state |START -- Monitoring for object reference changes and collecting statistics such as the object begin count. MARK -- Determining which objects should be collected and which should not. PAUSE -- Determining if any marked objects should not be collected. MARK COMPLETED -- Stops checking for reference changes (finalizing marked object list). DELETE -- Deleting objects. COMPLETE -- Completed cycle.| |Start time |The date and time the cycle began |Date and time stamp (local server time) | |Begin count |The total number of shared objects held by the server |Integer counter | |Paused stage |The total time the DGC paused |Milliseconds | |Mark stage |The total time the DGC took to mark objects for collection |Milliseconds | |Garbage count |The number of shared objects marked for collection |Integer counter | |Delete stage |The total time the DGC took to collect marked objects |Milliseconds | |Total elapsed time |The total time the DGC took to pause, mark objects, and collect marked objects |Milliseconds | The DGC graph combines a real-time line graph (with history) displaying the DGC total elapsed time with a bar graph showing the total number of freed objects. h4. Live Object Count If the trend for the total number of live objects goes up continuously, clients in the cluster will eventually run out of memory and applications may fail. Upward trends indicate a problem with application logic, garbage collection, or a tuning issue on one or more clients. The total number of live objects is given in the graph's title. !Terracotta Developer Console^ConsoleLiveObjects.png|thumbnail, vspace=8! DGC cycles should occur often enough to keep the total number of live objects from continuing to grow. h4. Triggering a DGC Cycle The DGC panel displays a message stating the configured frequency of DGC cycles. To manually trigger a DGC cycle, click *Run DGC*. h3. Operator Events The Operator Events panel displays cluster events received by the Terracotta server array. You can use the Operator Events panel to quickly view these events in one location in an easy-to-read format, without having to search the Terracotta logs. Events are listed in a table with the following columns: * Event Type - The level of the event (INFO, WARN, DEBUG, ERROR, CRITICAL) along with a color-coded light corresponding to the severity of the event. * Time of Event - The event's date and time stamp. * Node - The server receiving the event. * Event System - The Terracotta subsystem that generated the event. The choices are MEMORY MANAGER (virtual memory manager), DGC (distributed garbage collector), LOCK MANAGER (cluster-wide locks manager), and HA (server array). * Message - Message text reporting on a discrete event. An event appears in bold text until it is manually selected (highlighted). The text of an event that has been selected is displayed in regular weight. The Operator Events panel has the following controls: * Mark All Viewed - Click this button to change the text of all listed events from bold to regular weight. * Export - Click this button to export a text file containing all listed events. * Select View - Use this drop-down menu to filter the list of displayed events. You can filter the list of events based on type (level) or by the generating system (or subsystem). For example, if you choose INFO from the menu, only events with this event type are displayed in the event list. h3. Thread Dumps You can get a snapshot of the state of each server and client in the Terracotta cluster using thread dumps. To generate a thread dump, click *Take Thread Dump*. !tocDiagnosticsThreadDump.png|thumbnail, vspace=8! The thread-dump navigation pane lists completed thread dumps by date-time stamp. The contents of selected thread dumps are displayed in the right-side pane. A *Find* tool is available for searching through the currently displayed thread dump. To delete a thread dump (or all thread dumps) from the thread-dump navigation pane, right-click the thread dump in the thread-dumps navigation pane, then choose *Delete* (or *Delete All*) from the context menu. You can also delete a selected thread dump with your keyboard's Delete key. {note} Servers that appear in the console but are not running produce empty thread dumps. {note} h4. Saving Thread Dumps Thread dumps are deleted each time the console is restarted. To save all existing thread dumps to disk as a ZIP file, click *Export All...*. To save any thread dump as a text file, right-click the thread dump in the thread-dumps navigation pane, then choose *Export As Text...* from the context menu. h2. Logging The Logging panel displays log messages for each server. From the *View log for* menu, choose the server whose logs you want to view. !tocLogging.png|thumbnail, vspace=8! To open the Logging panel, click *Logging* in the control panel. You can select and, using the *Copy* command from the context menu, copy any portion of the log. The context menu also offers *Select All* and *Clear* commands. {div9}

Anchor
restore
restore

Restoring a Backup

Terracotta maintains a copy of shared in-memory data on disk. In most server-failure cases, Terracotta automatically restores that shared data by loading it from the copy, recreating the application state.

However, if you encounter a situation in which the data files are missing, you can restore them from backups.

To restore data files from a backup:

  1. Shut down the Terracotta cluster.
  2. Make copy of any existing data files.
  3. Delete the existing data files.
  4. Copy the backup data files to the directory from which you deleted the original (existing) data files.
  5. Restart the Terracotta cluster.

Diagnostics

The Diagnostics panels offers tools for monitoring and analysis. There are three tabbed panels available: one for monitoring and triggering the Terracotta Distributed Garbage Collector (DGC), one for viewing cluster events, and one for taking thread dumps.

Image Added

Tip
titleOperator Events

The cluster events are shown under the Operator Events panel. The button for accessing this panel, which is not shown in the screenshot above, is labeled Operator Events and appears between the Distributed GC and Thread Dump buttons.

To open the Diagnostics panels, click Diagnositics in the control panel.

Anchor
DistributedGarbageCollection
DistributedGarbageCollection

Distributed Garbage Collection

Objects in a DSO root object graph may become unreferenced and no longer exist in the Terracotta client's heap. These objects are eventually marked as garbage in a Terracotta server instance's heap and from persistent storage by the DGC. The DGC is unrelated to the Java garbage collector.

Info

For more information on the DGC, see the Terracotta Concept and Architecture Guide.

To view a history table of DGC activity in the current cluster, click Distributed garbage collection in the cluster list. The history table is automatically refreshed each time a collection occurs. Each row in the history table represents one distributed garbage collection cycle, with the following columns:

Column

Definition

Values

Iteration

The index number of the DGC cycle

Sequential integer

Type

The type of cycle

Full – Running a full collection cycle targeting all eligible objects.
Young – Running a collection cycle targeting Young Generation objects.

Status

The collection cycle's current state

START – Monitoring for object reference changes and collecting statistics such as the object begin count.
MARK – Determining which objects should be collected and which should not.
PAUSE – Determining if any marked objects should not be collected.
MARK COMPLETED – Stops checking for reference changes (finalizing marked object list).
DELETE – Deleting objects.
COMPLETE – Completed cycle.

Start time

The date and time the cycle began

Date and time stamp (local server time)

Begin count

The total number of shared objects held by the server

Integer counter

Paused stage

The total time the DGC paused

Milliseconds

Mark stage

The total time the DGC took to mark objects for collection

Milliseconds

Garbage count

The number of shared objects marked for collection

Integer counter

Delete stage

The total time the DGC took to collect marked objects

Milliseconds

Total elapsed time

The total time the DGC took to pause, mark objects, and collect marked objects

Milliseconds

The DGC graph combines a real-time line graph (with history) displaying the DGC total elapsed time with a bar graph showing the total number of freed objects.

Live Object Count

If the trend for the total number of live objects goes up continuously, clients in the cluster will eventually run out of memory and applications may fail. Upward trends indicate a problem with application logic, garbage collection, or a tuning issue on one or more clients. The total number of live objects is given in the graph's title.

Image Added

DGC cycles should occur often enough to keep the total number of live objects from continuing to grow.

Triggering a DGC Cycle

The DGC panel displays a message stating the configured frequency of DGC cycles. To manually trigger a DGC cycle, click Run DGC.

Operator Events

The Operator Events panel displays cluster events received by the Terracotta server array. You can use the Operator Events panel to quickly view these events in one location in an easy-to-read format, without having to search the Terracotta logs.

Events are listed in a table with the following columns:

  • Event Type - The level of the event (INFO, WARN, DEBUG, ERROR, CRITICAL) along with a color-coded light corresponding to the severity of the event.
  • Time of Event - The event's date and time stamp.
  • Node - The server receiving the event.
  • Event System - The Terracotta subsystem that generated the event. The choices are MEMORY MANAGER (virtual memory manager), DGC (distributed garbage collector), LOCK MANAGER (cluster-wide locks manager), and HA (server array).
  • Message - Message text reporting on a discrete event.

An event appears in bold text until it is manually selected (highlighted). The text of an event that has been selected is displayed in regular weight.

The Operator Events panel has the following controls:

  • Mark All Viewed - Click this button to change the text of all listed events from bold to regular weight.
  • Export - Click this button to export a text file containing all listed events.
  • Select View - Use this drop-down menu to filter the list of displayed events. You can filter the list of events based on type (level) or by the generating system (or subsystem). For example, if you choose INFO from the menu, only events with this event type are displayed in the event list.

Thread Dumps

You can get a snapshot of the state of each server and client in the Terracotta cluster using thread dumps. To generate a thread dump, click Take Thread Dump.

Image Added

The thread-dump navigation pane lists completed thread dumps by date-time stamp. The contents of selected thread dumps are displayed in the right-side pane. A Find tool is available for searching through the currently displayed thread dump.

To delete a thread dump (or all thread dumps) from the thread-dump navigation pane, right-click the thread dump in the thread-dumps navigation pane, then choose Delete (or Delete All) from the context menu. You can also delete a selected thread dump with your keyboard's Delete key.

Note

Servers that appear in the console but are not running produce empty thread dumps.

Saving Thread Dumps

Thread dumps are deleted each time the console is restarted.

To save all existing thread dumps to disk as a ZIP file, click Export All....

To save any thread dump as a text file, right-click the thread dump in the thread-dumps navigation pane, then choose Export As Text... from the context menu.

Logging

The Logging panel displays log messages for each server. From the View log for menu, choose the server whose logs you want to view.

Image Added

To open the Logging panel, click Logging in the control panel.

You can select and, using the Copy command from the context menu, copy any portion of the log. The context menu also offers Select All and Clear commands.