Child pages
  • Terracotta Developer Console
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 24 Next »

About Terracotta Documentation

This documentation is about Terracotta DSO, an advanced distributed-computing technology aimed at meeting special clustering requirements.

Terracotta products without the overhead and complexity of DSO meet the needs of almost all use cases and clustering requirements. To learn how to migrate from Terracotta DSO to standard Terracotta products, see Migrating From Terracotta DSO. To find documentation on non-DSO (standard) Terracotta products, see Terracotta Documentation. Terracotta release information, such as release notes and platform compatibility, is found in Product Information.

Unknown macro: {div9}
Release: 3.6
Publish Date: November, 2011

Documentation Archive »

Unknown macro: {HTMLcomment}

UI stuff:

  • server info:
    • Main: doesn't tell user location of tc-config.xml being used
    • Main: says "Failover mode disk-based-active-passive" even with only 1 server in cluster
    • Environment: "tc.config: tc-config.xml" but doesn't seem to indicate location
  • where to monitor:
    • Server runtime statistics
  • Cluster "recording on" light should be doc in Lock Profiler and Statistics Recorder, and where ever else it's triggered from

Terracotta Developer Console

Introduction

The Terracotta Developer Console delivers a full-featured monitoring and diagnostics tool aimed at the development and testing phases of an application clustered with Terracotta. Use the Developer Console to isolate issues, discover tuning opportunities, observe application behavior with clustering, and learn how the cluster holds up under load and failure conditions. The console functions as a JMX client with a graphical user interface.

Enterprise versions of Terracotta include the Terracotta Operations Center, a GUI operator's console offering features such as backups of shared data, client disconnect, and server shutdown controls. To learn more about the many benefits of an enterprise version of Terracotta, see our [Enterprise Products|enterprise:Products].

Using the console, you can:

  • Monitor the health of servers and clients under changing conditions.
  • Force distributed garbage collection operations
  • View cluster-wide statistics
  • Record cluster statistics for later display
  • Receive console status and server log messages in a static window
  • Learn exactly which classes are being shared in the cluster

These and other console features are described below.

Some statistics available through the Terracotta Developer Console are abouts Terracotta transactions. Terracotta transactions are not application transactions. One Terracotta transaction is a batch of one or more writes to shared data. See the Concept and Architecture Guide for more information.

Launching the Terracotta Developer Console

You can launch the Terracotta Developer Console from a command line.

The console can also be launched from the Terracotta Welcome application. If you are new to Terracotta, [take the tour] to learn how to download Terracotta and use the Welcome application.

Microsoft Windows

[PROMPT] %TERRACOTTA_HOME%\bin\dev-console.bat

UNIX/Linux

[PROMPT] ${TERRACOTTA_HOME}/bin/dev-console.sh&

What You Will See

When not connected to a server, the console displays a connect/disconnect panel, message-log window, status line, and an inactive cluster node in the clusters panel.

null

The cluster list in the cluster panel may already be populated because of pre-existing references to previously defined Terracotta clusters. These references are maintained as Java properties and persist across sessions and product upgrades. If no clusters have been defined, a default cluster (host=localhost, jmx-port=9520) is created.

Setting a JMX Port

To learn more about setting JMX ports, see the Configuration Guide and Reference.

Once the console is connected to a cluster, the cluster node in the cluster panel serves as the root of an expandable/collapsible tree with nested displays and controls. More than one cluster node can appear in the cluster panel.

Menus

The following menus are available from the console's menu bar.

  • File
    • New Cluster – Creates a new cluster node in the cluster panel to allow the console to separately connect to another cluster. Has no effect outside of the console itself.
    • Quit – Shut down the console application. Has no effect on the cluster except to reduce load if the console has been recording statistics or profiling locks.
  • Tools
    • Show SVT – Opens the Terracotta Snapshot Visualizer or takes you to the download page.
    • Options – Opens the Options dialog.

File

The File men

Context-Sensitive Help

Context-sensitive help is available wherever null (help button) appears in the Terracotta Developer Console. Click null in a console panel to open a web-browser page containing help on the features in that panel.

Working with Clusters

Clusters are the highest-level nodes in the expandable cluster list displayed by the Terracotta Developer Console. A single Terracotta cluster defines a domain of Terracotta servers and clients being clustered by Terracotta. A single Terracotta cluster can have one or more servers and one or more clients. For example, the Terracotta servers in a setup featuring server redundancy, along with their clients, appear under the same cluster.

Adding and Removing Clusters

To add a new cluster reference, choose File|New cluster. Or right-click in the cluster list area to open a context menu, then choose New cluster.

The context menu is usually opened by right-clicking an item.

The cluster topology is determined from the server specified in the connection panel's Server Host and JMX Port fields. These fields are editable when the console is not connected to the cluster.

To remove an existing cluster reference, right-click the cluster in the cluster list to open the context menu and choose Delete.

Connecting to a cluster

To connect to an existing cluster, select the cluster node in the cluster list, then click the Connect button in the connect/disconnect panel. You can also connect to a specific cluster by choosing Connect from its context menu. After a successful connection, the cluster node becomes expandable and server connection information appears in the connect/disconnect panel.

null

To automatically connect to a cluster whenever the Terracotta Developer Console starts or when at least one of the cluster's servers is running, enable Auto-connect in the cluster context menu. Automatic connections are attempted in the background and do not interfere with normal console operation.

Connecting to a Secured Cluster

A Terracotta cluster can be secured for JMX access, requiring authentication before access is granted. Connecting to a secured cluster prompts users to enter a username and password.

null

For instructions on how to secure your Terracotta cluster for JMX, see the Configuration Guide and Reference.

Disconnecting from a Cluster

To disconnect from a cluster, click Disconnect on the cluster panel or by select Disconnect from the cluster context menu.

Working with Servers

The Servers node appears under a connected Terracotta cluster in the cluster list. The Servers panel displays a table of servers in the cluster, including each server's name, hostname or IP address, and JMX port.

null

To view the servers in a cluster, expand the Terracotta cluster, then expand the Servers node.

Unknown macro: {enterprise-feature}

Pending Client Transactions

The Servers panel in the enterprise version of the Terracotta Developer Console displays a bar graph showing the number of unacknowledged Terracotta transaction broadcasts for each client in the cluster.

null

You can open a client's panel by double-clicking its bar graph in the Servers panel. This is useful for quickly locating and investigating clients that maintain a large or growing number of unacknowledged Terracotta transaction broadcasts. Such clients could slow down an entire cluster and may need to be disconnected.

Server Panel

Selecting a specific server's node displays that server's panel, with the Main, Environment, and Config tabs.

The Main tab displays the server status and a list of properties, including the server's IP address, version, license, and persistence and failover modes.

null

The Environment tab displays the server's JVM system properties and provides a case-sensitive Find tool.

null

The Config tab displays the Terracotta configuration the server is using and provides a case-sensitive Find tool.

null

Each server node contains two child nodes: Runtime statistics and Thread dumps.

Connecting and Disconnecting from a Server

The Terracotta Developer Console is connected to all of a cluster's servers when it's connected to the cluster. Being connected to a server means that the console is listening for JMX events coming from that server.

If you have confirmed that a Terracotta server is running, but the Terracotta Developer Console is unable to connect to it, a firewall on your network may be blocking the server's JMX port.

The console is disconnected from a cluster's servers when it's disconnected from the cluster. The console is also disconnected from a server when that server is shut down, but the server still appears as part of the cluster. However, its connection status changes.

Servers can be shut down using the [stop-tc-server script] or the server shutdown button (see below).

Unknown macro: {enterprise-feature}
The Server Shutdown Button

null
To shut down a Terracotta server:

  1. Select the server in the Servers branch of the Terracotta cluster list.
  2. Click Main to display the server's main panel.
  3. Click Shutdown.
    A dialog prompts to you confirm the action.

Server Connection Status

A Terracotta server's connection status is indicated by a status light next to the server's name. The light's color indicates the server's current connection status. A cluster can have one server, or be configured with multiple servers that communicate state over the network or use a shared file-system.

The following figure shows a two-member cluster with an active server named primary and a passive server named secondary in standby mode.

null

The following table summarizes the connection status lights.

Status Light

Server Status

Notes

Green

Active

The server is connected and ready for work.

Red

Unreachable

The server, or the network connection to the server, is down.

Yellow

Starting or Standby

A server is starting up; in a disk-based multi-server cluster, a passive server goes into standby mode until a file lock held by the active server is released.

Orange

Initializing

In a network-based multi-server cluster, a passive server must initialize its state before going into standby mode.

Cyan

Standby

In a network-based multi-server cluster, a passive server is ready to become active if the active server fails.

Working with Clients

The Clients node appears under a connected Terracotta cluster in the cluster list. The *Clients panel displays a table of connected clients. The table has the following columns:

  • Host - The client machine's hostname.
  • Port - The client's DSO Port.
  • ChannelID - The client's channel ID number.
  • Live Objects - The number of shared objects currently in the client's heap.

null

To view the list of clients in a cluster, expand the Terracotta cluster, then expand the Clients node.

Client Panel

When a Terracotta client connects to the cluster being monitored by the Terracotta Developer Console, a client node is created under the Clients node.

Selecting a specific client's node displays that client's panel, with the Main, Environment, Config, and Logging tabs.

The Main tab displays a list of client properties such as hostname and DSO port.

null

The Environment tab displays the client's JVM system properties and provides a case-sensitive Find tool.

null

The Config tab displays the Terracotta configuration the client is using and provides a case-sensitive Find tool.

null

The Logging tab displays options to add logging items corresponding to DSO client debugging. See the Configuration Guide and Reference for details on the various debug logging options.

null

Connecting and Disconnecting Clients

When started up properly, a Terracotta client is automatically added to the appropriate cluster.

When a Terracotta client is shut down or disconnects from a server, that client is automatically removed from the cluster and no longer appears in the Terracotta Developer Console cluster list.

Unknown macro: {enterprise-feature}

The Client Disconnect Button

null
To shut down a Terracotta client:

  1. Select the client in the Clients branch of the Terracotta cluster.
  2. Click Main to display the client's main panel.
  3. Click Disconnect.
    A dialog prompts to you confirm the action.

Monitoring Clusters, Servers, and Clients

The Terracotta Developer Console provides visual monitoring functions using icons, graphs, statistics, counters, and both simple and nested lists. You can use these features to monitor the overall health of your cluster as well as the health of individual cluster components.

Shared Objects

Applications clustered with Terracotta use shared objects to keep data coherent. Monitoring shared objects serves as an important early-warning and troubleshooting method that allows you to:

  • Confirm that appropriate object sharing is occurring;
  • be alerted to potential memory issues;
  • learn when it becomes necessary to tune garbage collection;
  • locate the source of object over-proliferation.

The Terracotta Developer Console provides the following tools for monitoring shared objects:

These tools are discussed in the following sections.

Cluster Object Browsers

The cluster object browser is a panel displaying the shared object graphs for an entire cluster. All of the shared objects in the cluster-wide heap are graphed, but the browser doesn't indicate which clients are sharing them. To see object graphs specific to a client, see the client object browser.

null

To open the cluster object browser, click Cluster object browser under a cluster node. The browser panel displays a running total of the live objects in the cluster. This is the number of objects currently found in the cluster-wide heap; however, this total does not correspond to the number of objects you see in the object graph because certain objects, including literals such as strings, are not counted. These uncounted objects appear in the object graph without an object ID.

Each element in an object graph has the following format:

fully.qualified.name.of.object (fully.qualified.name.of.object.type) [<number of subelements shown>/<total number of subelements>] [@<object ID]

The following are important aspects of the object graph display:

  • The top-level objects in an object graph correspond to the shared roots declared in the Terracotta server's configuration file (tc-config.xml).
    Root objects are indicated by a yellow diamond icon.
  • Objects in the graph that are of a collections type, and reference other objects, appear as a parent node.
    By clicking the triangular buttons next to their names, you can expand/collapse parent objects to display/hide child objects.
  • By default, up to ten children (fields) are displayed when you expand a collections-type object in the object graph.
    Right-click a collections-type object to open a context menu and set the number of children displayed when that object is expanded.

    A delay may occur when the object browser attempts to display very large graphs.

  • An object linked to a parent object is indicated by an up-arrow instead of a yellow diamond for a bullet.

An object browser does not refresh automatically. You can refresh it manually in any of the following ways:

  • Expand or collapse any part of any object graph.
  • Press the F5 key on your keyboard.
  • Right-click on any item on the object graph and select Refresh.

Client Object Browsers

The client object browser is a panel displaying the shared object graphs for a single client. All of the shared objects known to the client are graphed, but the ones not being shared by the client are grayed out.

null

To open the client object browser, click Client object browser under a client node. The browser panel displays a running total of the live objects in the client. This is the number of objects currently found in the client heap; however, this total does not correspond to the number of objects you see in the object graph because the following types of objects are not counted:

  • Objects not being shared by the client
    These unshared objects are grayed out in the object-browser view.
  • Literals such as strings
    These objects appear in the object graph without an object ID.

Each element in an object graph has the following format:

fully.qualified.name.of.object (fully.qualified.name.of.object.type) [<number of subelements shown>/<total number of subelements>] [@<object ID]

The following are important aspects of the object graph display:

  • The top-level objects in an object graph correspond to the shared roots declared in the Terracotta server's configuration file (tc-config.xml).
    Root objects are indicated by a yellow diamond icon.
  • Objects in the graph that are of a collections type, and reference other objects, appear as a parent node.
    By clicking the triangular buttons next to their names, you can expand/collapse parent objects to display/hide child objects.
  • By default, up to ten children (fields) are displayed when you expand a collections-type object in the object graph.
    Right-click a collections-type object to open a context menu and set the number of children displayed when that object is expanded.

    A delay may occur when the object browser attempts to display very large graphs.

  • An object linked to a parent object is indicated by an up-arrow instead of a yellow diamond for a bullet.

An object browser does not refresh automatically. You can refresh it manually in any of the following ways:

  • Expand or collapse any part of any object graph.
  • Press the F5 key on your keyboard.
  • Right-click on any item on the object graph and select Refresh.

Classes Browser

DSO allows for transparent, clustered object state synchronization. To accomplish this feature, some of your application classes will be adapted into new classes that are cluster-aware. The set of all such adapted classes known to the server are displayed in the Classes panel. The Tabular tab show all the adapted classes in a spreadsheet view, including the class name and a count of the number of instances of the class that have been created since the server started. The Tree tab shows a hierarchical, or Java package, view of the adapted classes. Finally, the TreeMap tab shows a presentation making it easy to quickly distinguish the most (and least) heavily used adapted classes. These views are a snapshots of the adapted classes known to the server. You can refresh these values by selecting the Refresh context menu on the Classes node.

null

Object Flush and Fault Rate Graphs

Object flush and fault rates are a measure of shared data flow between Terracotta servers and clients. These graphs can reflect trends in the flow of shared objects in a Terracotta cluster. Upward trends in flow can indicate insufficient heap memory, poor locality of reference, or newly changed environmental conditions. For more information, see Object Flush Rate and Object Fault Rate.

Cache Miss Rate Graph

The Cache Miss Rate measures the number of client requests for an object that cannot be met by a server's cache and must be faulted in from disk. An upward trend in this graph can expose a bottleneck in your cluster. For more information, see Cache Miss Rate.

Runtime Logging of New Shared Objects

You can log the creation of all new shared objects by following these steps:

  1. Select the target client in the cluster list.
  2. Click the Logging tab.
  3. Enable New object debug from the Runtime list.

During development or debugging operations, logging new objects may reveal patterns that introduce inefficiencies or errors into your clustered application. However, during production it is recommended that this type of intensive logging be disabled.

See the Configuration Guide and Reference for details on the various debug logging options.

Runtime Statistics

Runtime statistics provide a continuous feed of sampled real-time data on a number of server and client metrics. The data is plotted on a graph with configurable polling and historical periods. Sampling begins automatically when a runtime statistic panel is first viewed, but historical data is not saved. To record and save historical data, see the Cluster Statistics Recorder.

null

Each runtime statistics panel has the following controls:

  • Poll period seconds – Sets how often the console polls the server or client for data.
  • History Minutes – Sets the number of minutes of data shown on each metric's graph.
  • null – Starts data polling again (reset Pause).
  • null – Pauses data polling and graphing.
  • null – Deletes existing historical data from the graphs.

To view runtime statistics for an individual server or client, click Runtime statistics under that server or client in the cluster list.

Unknown macro: {enterprise-feature}

Cluster Runtime Stats

To view runtime statistics for every server and client in a cluster in one panel, click Cluster runtime stats under that cluster in the cluster list.

null

Specific runtime statistics are defined in the following sections. The cluster components for which the statistic is available are indicated in parentheses.

Heap Usage (Cluster, Server, Client)

Shows the amount, in megabytes, of maximum available heap and heap being used.

CPU Usage (Cluster, Server, Client)

Shows the CPU load as a percentage. If more than one CPU is being used, each CPU's load is shown in a separate graph line.

Transaction Rate (Cluster, Server, Client)

Shows the number of completed Terracotta transactions. Terracotta transactions are sets of one or more clustered object changes, or writes, that must be applied atomically.

Cache Miss Rate (Server)

Disk-to-server faults occur when an object is not available in a server's in-memory cache. The Cache Miss Rate statistic is a measure of how many objects (per-second) are being faulted from the disk in response to client requests. Objects being requested for the first time, or objects that have been flushed from the server heap before a request arrives, must be faulted in from disk. A high Cache Miss Rate may indicate inadequate memory allocation at the server.

Unacknowledged Transaction Broadcasts (Client)

Every Terracotta transactions in a Terracotta cluster must be acknowledged by Terracotta clients with in-memory shared objects that are affected by that transaction. For each client, Terracotta server instances keep a count of transactions that have not been acknowledged by that client. The Unacknowledged Transaction Broadcasts statistic is a count of how many transactions the client has yet to acknowledge. An upward trend in this statistic indicates that a client is not keeping up with transaction acknowledgments, which can slow the entire cluster. Such a client may need to be disconnected.

Object Flush Rate (Server, Client)

The Object Flush Rate statistic is a measure of how many objects are being flushed out of client memory to the Terracotta server. These objects are available in the Terracotta server if needed at a later point in time. A high flush rate may indicate inadequate memory allocation at the client.

On a server, the Object Flush Rate is a total including all clients. On a client, the Object Flush Rate indicates only the objects that client is flushing.

Object Fault Rate (Server, Client)

The Object Fault Rate statistic is a measure of how many objects are being faulted into client memory from the server. A high fault rate may indicate poor locality of reference or inadequate memory allocation at the client.

On a server, the Object Fault Rate is a total including all clients. On a client, the Object Fault Rate indicates only the objects that have been faulted to it.

When the Terracotta server faults an object, it also faults metadata for constructing a certain number of the objects referenced by or related to that object. This improves locality of reference. See the definition of the fault-count property in Terracotta Configuration Guide and Reference for more information.

Lock profiler

The Terracotta runtime system can be directed to gather statistics about the distributed locks that are realized as a result of the configuration you have created. Using these statistics can give you insight into the nature of your DSO application, letting you discover the code paths leading to highly-contented access to shared state.

The Lock Profiler panel lets you enable/disable lock statistics gathering, specify the lock code path trace depth for those statistics, and query the server to refresh the stats display.

Gathering and recording statistics may impact a cluster's performance. If statistics are being gathered, you are alerted in the cluster list by a flashing icon next to the affected cluster.

Getting started with lock statistics:

  • Navigate to the Locks stats panel by clicking the Lock stats node in the main navigation tree
  • Enable statistics gathering by clicking the On button
  • Optionally set the desired trace depth (default=0)
  • Wait some period of time for some stats to be gathered and transmitted to the server
  • Click the Refresh button to update the display

The Locks Profiler panel is comprised of the Client and Server stats pages.

null

The client locks stats are based on the code paths in your DSO client applications that resulted in a lock being taken out. The Server page stats concern the cluster-wide nature of the distributed locks. Each lock has a corresponding identifier, the lock-id. For a named lock the lock-id is the lock name. For an autolock the lock-id is the server-generated id of the object on which that lock was taken out. An example of an autolock id is @1001. That autolock id corresponds to the shared object upon which distributed synchronization was carried out. You can use the object browser to view the state of shared object @1001.

A single lock-expression in the configuration can result in the creation of multiple locks by the use of wildcard patterns. A single lock may be arrived at through any number of different code paths. A trace depth of zero means that you don't care how the lock was arrived at, but by increasing the trace depth the client-side lock stats are broken out by the corresponding number of Java call-stack frames. For example, there may be 3 different call sequences that resulted in a particular lock being granted, with one of the paths rarely entered and another responsible for the majority of those lock grants. By setting the trace depth appropriately you can gain insight into the behavior of your application and how it may affect the performance of your clustered system. Note that trace stack frames will only include Java source line numbers if the code was compiled with debugging enabled, commonly done by passing the javac command the -g flag and in Ant by defining the javac task with the debug="true" attribute.

The Main Controls

Enable Lock Profiling

Use this control to turn lock profiling on or off.

Trace Depth

The trace depth control sets the number of client call-stack frames that are analyzed per lock event to record lock statistics. A depth of 0 gathers lock statistics without regard to how the lock event was arrived at.

A lock depth of 1 means that one call-stack frame will be used to disambiguate different code paths when the lock event occurred. A lock depth of 2 will use two frames, and so on. So for example, with a setting of 1, all locks will appear be recorded together, regardless of the call path, because the stack depth analyzed will always be just the method that resulted in the lock event to occur (in other words the surrounding method). However, beginning with a lock depth of 2, different call paths can be separated, since not only the surrounding method, but also the calling method, will be used to record different lock statistics.

This means that for example, when the trace depth setting is set to 1, a lock event that occurs within method Foo() will record all lock events occurring within Foo() as one single statistic, e.g. the number of lock requests, average held time and so on. But, with a trace depth setting of 2, the callers of Foo(), say Bar1() and Bar2(), will also be considered, so that a call path of Bar1() -> Foo() will be recorded separately from Bar2() -> Foo().

Refresh

Use this button to get an updated view of the statistics gathered so far.

The Data Table


Lock Statistic

Description

Times Requests

Number of times this lock was requested

Times Hopped

Times an acquired greedy lock was retracted from holding client and granted to another

Average Contenders

Average number of threads wishing to acquire this lock at the time it was requested

Average Acquire Time

Average time between lock request and grant

Average Held Time

Average time grantee held this lock

Average Nested Lock Depth

Average number of outstanding locks held by acquiring thread at grant time


Lock Element Details

The bottom portion of the Clients page displays detail on the selected lock element. The currently selected lock trace is shown on the left and the configuration element responsible for the creation of the selected lock is shown on the right.

A special note has to be made with respect to the Server lock stats. The Terracotta system employs the concept of greedy locks to help improve performance by limiting unnecessary lock hops. Once a client has been awarded a lock it is allowed to keep that lock until another client requests it. The assumption is that once a client obtains a lock it will likely want to request that same lock again shortly. So a lock is handed out to a client until such a time that a different client wants that lock. So if your cluster consisted of a single node that manipulated a single shared object repeatedly, the Server lock requests would be 1 until another client entered the cluster and began manipulating that object. It is likely that when you see Server stats that are undefined (na) it is due to the nature of greedy locks.

Distributed Garbage Collection

Objects in a DSO root object graph may become unreferenced and no longer exist in the Terracotta client's heap. These objects are eventually marked as garbage in a Terracotta server instance's heap and from persistent storage by the Terracotta Distributed Garbage Collector (DGC). The DGC is unrelated to the Java garbage collector.

For more information on the DGC, see the Terracotta Concept and Architecture Guide.

To view a history table of DGC activity in the current cluster, click Distributed garbage collection in the cluster list. The history table is automatically refreshed each time a collection occurs. Each row in the history table represents one distributed garbage collection cycle, with the following columns:

Column

Definition

Values

Iteration

The index number of the DGC cycle

Sequential integer

Type

The type of cycle

Full – Running a full collection cycle targeting all eligible objects.
Young – Running a collection cycle targeting Young Generation objects.

Status

The collection cycle's current state

START – Monitoring for object reference changes and collecting statistics such as the object begin count.
MARK – Determining which objects should be collected and which should not.
PAUSE – Determining if any marked objects should not be collected.
MARK COMPLETED – Stops checking for reference changes (finalizing marked object list).
DELETE – Deleting objects.
COMPLETE – Completed cycle.

Start time

The date and time the cycle began

Date and time stamp (local server time)

Begin count

The total number of shared objects held by the server

Integer counter

Paused stage

The total time the DGC paused

Milliseconds

Mark stage

The total time the DGC took to mark objects for collection

Milliseconds

Garbage count

The number of shared objects marked for collection

Integer counter

Delete stage

The total time the DGC took to collect marked objects

Milliseconds

Total elapsed time

The total time the DGC took to pause, mark objects, and collect marked objects

Milliseconds

Thread Dumps

You can get a snapshot of the state of each server and client in the Terracotta cluster using the Terracotta Developer Console thread dumps feature.

null

Every Terracotta cluster, server, and client in the cluster list has a thread-dumps panel. Each panel has a list logging completed thread dumps, and a pane to display the contents of a selected thread dump. Each panel also provides a case-sensitive Find tool for searching through the currently displayed thread dump.

Cluster Thread Dumps

null

To generate, view, and save a cluster-wide thread dump:

  1. Expand the target cluster in the cluster list.
  2. Click Cluster thread dumps to open the thread-dumps panel.
  3. Click Take thread dump.
    The thread dump is logged in an expandable list.
  4. Expand an entry in the thread-dumps log to see the thread-dump subentries for each server and client.
  5. Click a server or client subentry to display the thread dump for that server or client.
  6. To save a cluster-wide thread dump as a zip file, click Export....

Server Thread Dumps

null

To generate and view a thread dump for any server process:

  1. Expand the target server in the cluster list.
  2. Click Server thread dumps to open the thread-dumps panel.
  3. Click Take thread dump.
    The thread dump is logged in a list.
    #Click the entry in the list to display the thread dump.

Client Thread Dumps

null

To generate and view a thread dump for any client process:

  1. Expand the target client in the cluster list.
  2. Click Client thread dumps to open the thread-dumps panel.
  3. Click Take thread dump.
    The thread dump is logged in a list.
    #Click the entry in the list to display the thread dump.

Recording and Viewing Statistics

Cluster Statistics Recorder

The Cluster Statistics Recorder panel can generate recordings of selected cluster-wide statistics. This panel has controls to start, stop, view, and export recording sessions. You can use the Snapshot Visualization Tool to view the recorded information.

null

For definitions of available statistics Cluster Statistics Definitions. To learn about configuring the Statistics Recorder, using its command-line interface, and more, see [The Terracotta Cluster Statistics Recorder].

Gathering and recording statistics may impact a cluster's performance. If statistics are being gathered, you are alerted in the cluster list by a flashing icon next to the affected cluster.

">Snapshot Visualization Tool

The Snapshot Visualization Tool (SVT) provides a graphical view of cluster information and statistics. The view is created using data recorded with the Statistics Recorder.

null

The SVT is provided as a TIM, called tim-svt, which you can install from the TIM Update Center. Once the SVT is installed, start (or restart) the Terracotta Developer Console and confirm that the View button on the Cluster Statistics Recorder is enabled. No changes to the Terracotta configuration file are necessary when you install the SVT.

In Eclipse with the Terracotta DSO Eclipse plug-in, use the Terracotta|Update modules... menu to install SVT.

SVT controls are defined below.

Import...

Load a saved cluster statistics recording that was saved to file. Clicking Import... opens a standard file-selection window for locating the file.

Retrieve...

Find recorded sessions on active Terracotta servers. Clicking Retrieve... opens a dialog to enter the server address in the format <server_ip_address:JMX_port> or <server_name:JMX_port> as defined in tc-config.xml. Once connected to the server, available recorded sessions appear in the SVT Sessions menu.

Sessions

Select a recorded session from the Sessions drop-down menu to load it. Sessions lists the recorded sessions last retrieved from a Terracotta server.

Graph heights

Scale the y-axis on the graphs displayed in the SVT. Moving the slider to the left shrinks the graph height, while moving it to the right grows the graph height.

Server, Client, and Statistics checkboxes

Choose which servers and clients have their recorded statistics displayed in the graphs. For each server and client, choose which statistics are displayed in the graphs.

Unknown macro: {enterprise-feature}

Backing Up Shared Data

Using the Terracotta backup feature, you can create a backup of the data being shared by your application.

To access the backup feature, choose Backup database for the cluster whose servers you want to back up. The Backup database panel appears.

null

To perform a backup, click Backup DB. A dialog box appears where you can confirm the backup destination directory or enter a new destination. The backup is saved to the directory objectdb at the destination. The database is always backed up to a directory called objectdb, which is automatically created if it does not exist at the destination.

Backups can also be initiated using the [backup-data script]. The script is useful for automating scheduled backups.

To change the default backup directory path, edit the <data-backup> property in the Terracotta server's configuration file with the path to your preferred backup directory:

<server>
<data-backup>path/to/my/backup/directory</data-backup>
</server>

Enabling Backups

If a Terracotta server is not configured for permanent-store persistence, the Backup DB button is disabled and the following message appears on the Backup database panel:

Backup feature is currently disabled because the cluster is operating in {{temporary-swap-only}} persistence mode.

To enable the Backup DB button, change the value of the persistence mode property in the Terracotta server's configuration file to permanent-store:

<server>
   <dso>
      <persistence>
         <mode>permanent-store</mode>
      </persistence>
   </dso>
</server>

Restoring a Backup

Terracotta maintains a copy of shared in-memory data on disk. In most server-failure cases, Terracotta automatically restores that shared data by loading it from the copy and your application state is recreated.

However, if you encounter a situation in which the data files are missing, you can restore them from backups.

To restore data files from a backup:

  1. Shut down the Terracotta cluster.
  2. Make copy of any existing data files.
  3. Delete the existing data files.
  4. Copy the backup data files to the directory from which you deleted the original (existing) data files.
  5. Restart the Terracotta cluster.

Update Checker

On a bi-weekly basis the Terracotta Developer Console will check, when first started, on updates to the Terracotta platform. By default, a notice informing you that an update check is about to be performed is displayed, allowing to ignore the immediate check, acknowledge and allow the check, or to disable further checking.

null

Should the update check be allowed, the Terracotta Developer Console will query the OpenTerracotta website (www.terracotta.org) and report on any new updates. Should the update checker feature be disabled, it can always be re-enabled via the Help|Update Checker... menu.

Appendix: Definitions of Cluster Statistics

The following categories of cluster information and statistics are available for viewing and recording in the Terracotta Cluster Statistics Recorder.

l1 = Terracotta client
l2 = Terracotta server instance
See the Glossary for definitions of commonly used terms.

cache objects evict request

The total number of objects marked for eviction from the l1, or from the l2 to disk. Evicted objects are still referenced, and can be faulted back to the l1 from the l2 or from disk to l2. The SVT graphs this metric for each l1 and l2 separately. High counts imply that memory may be running low.

cache objects evicted

The number of objects actually evicted. If this metric is not close in value to cache objects evict request, then memory may not be getting freed quickly enough.

l1 l2 flush

The object flush rate when the l1 flushes objects to the l2 to free up memory or as a result of GC activity.

l2 faults from disk

The number of times the l2 has to load objects from disk to serve l1 object demand. A high faulting rate may indicate an overburdened l2.

l2 l1 fault

The number of times an l2 has to send objects to an l1 because the objects do not exist in the l1 local heap due to memory constraints. Better scalability is achieved when this number is lowered through improved locality and usage of an optimal number of JVMs.

memory (usage)

The amount of memory (heap) usage over time.

vm garbage collector

The standard Java garbage collector's behavior, tracked on all JVMs in the cluster.

distributed gc (distributed garbage collection, or DGC)

The behavior of the Terracotta tool that collects distributed garbage on an l2. The DGC can pause all other work on the l2 to ensure that no referenced objects are flagged for garbage collection.

l2 pending transactions

The number of Terracotta transactions held in memory by a Terracotta server instance for the purpose of minimizing disk writes. Before writing the pending transactions, the Terracotta server instance optimizes them by folding in redundant changes, thus reducing its disk access time. Any object that is part of a pending transaction cannot be changed until the transaction is complete.

stage queue depth

The depth to which visibility into Terracotta server-instance work-queues is available. A larger depth value allows more detail to emerge on pending task-completion processes, bottlenecks due to application requests or behavior, types of work being done, and load conditions. Rising counts (of items in these processing queues) indicate backlogs and may point to performance degradation.

server transaction sequencer stats

Statistics on the Terracotta server-instance transaction sequencer, which sequences transactions as resources become available while maintaining transaction order.

network activity

The amount of data transmitted and received by a Terracotta server instance in bytes per second.

l2 changes per broadcast

The number of updates to objects per broadcast message (see l2 broadcast count).

message monitor

The network message count flowing over TCP from Terracotta clients to the Terracotta server.

l2 broadcast count

The number of times that a Terracotta server instance has transmitted changes to objects. This "broadcast" occurs any time the changed object is resident in more than one Terracotta client JVM. This is not a true broadcast since messages are sent only to clients where the changed objects are resident.

l2 transaction count

The number of Terracotta transactions being processed (per second) by a Terracotta server instance.

l2 broadcast per transaction

The ratio of broadcasts to Terracotta transactions. A high ratio (close to 1) means that each broadcast is reporting few transactions, and implies a high co-residency of objects and inefficient distribution of application data. A low ratio (close to 0) reflects high locality of reference and better options for linear scalability.

system properties

Snapshot of all Java properties passed in and set at startup for each JVM. Used to determine configuration states at the time of data capture, and for comparison of configuration across JVMs.

thread dump

Displays a marker on all statistics graphs in the SVT at the point when a thread dump was taken using the Cluster statistics recorder. Clicking the marker displays the thread dump.

disk activity

The number of operations (reads and writes) per second, and the number of bytes per second (throughput). The number of reads corresponds to l2 faulting objects from disk, while writes corresponds to l2 flushing objects to disk. The SVT graphs the two aspects separately.

cpu (usage)

The percent of CPU resources being used. Dual-core processors are broken out into CPU0 and CPU1.

More Information

More information on the following topics is available in other Terracotta documentation:

  • No labels