Child pages
  • JMX Guide

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin
Include Page
docs:documentation-navbardocs:
documentation-navbar
pagetitle
Reference Documentation
Section
{div:style=
Column
width250px
Wiki Markup
260px
} {include:
Div
stylemargin-top:20px
Include Page
documentation-index
documentation-index
} {div}
Column
{div9:id=documentationcolumn} h1. JMX Guide
Wiki Markup
Div
iddocumentationcolumn
Include Page
HeaderBasic
HeaderBasic

JMX Guide

comment

TODO: http://jira.terracotta.org/jira/browse/DOC-178

The

Terracotta

server

publishes

a

number

of

MBeans

to

provide

information

on

the

operation

of

the

cluster.

The

[

Terracotta

Console|Admin Console Guide] uses the JMX interface to provide

Developer Console and the Terracotta Operations Center use the JMX interface to provide real-time

information,

but

the

same

interface

can

be

used

by

any

JMX

client.

As

of

Terracotta

2

3.

3

0,

there

is

also

a

new set of MBeans to report cluster events such as client nodes joining or leaving the cluster. {anchor:authentication} h2. JMX Authentication It is possible to configure the Terracotta Server to require authentication credentials before JMX communications take place. This will secure server shutdown, the Admin Console, JConsole, and any other processes attempting to connect via JMX. Server authentication will not provide secure message transmission once valid credentials are provided by a listening client. The JMX messages will not be encrypted. For this reason, we recommend that the Terracotta server be placed in a secure location on a private network and only be queried remotely via an encrypted tunnel such as those provided by stunnel or SSH. Configuring JMX authentication with Terracotta requires that an authentication element be placed in the Terracotta configuration XML file, read by the Terracotta server instance that requires security. Terracotta relies on the standard Java security mechanisms for JMX which involve the creation of a .access and .password file with correct permissions set. The default location for these files is in (for JDK 1.5+)

new API to report platform cluster events such as client nodes joining or leaving the cluster.

Anchor
authentication
authentication

JMX Authentication

It is possible to configure the Terracotta Server to require authentication credentials before JMX communications take place. This will secure server shutdown, the Terracotta Developer Console, the Terracotta Operations Center, JConsole, and any other processes attempting to connect via JMX. Server authentication will not provide secure message transmission once valid credentials are provided by a listening client. The JMX messages will not be encrypted. For this reason, we recommend that the Terracotta server be placed in a secure location on a private network and only be queried remotely via an encrypted tunnel such as those provided by stunnel or SSH.

Configuring JMX authentication with Terracotta requires that an authentication element be placed in the Terracotta configuration XML file, read by the Terracotta server instance that requires security. Terracotta relies on the standard Java security mechanisms for JMX which involve the creation of a .access and .password file with correct permissions set. The default location for these files is in (for JDK 1.5+) $JAVA_HOME/jre/lib/management.

Once

configured

properly,

the

Admin Console connection dialog will prompt you for credentials before completing a server connection. The

Terracotta Developer Console (or the Terracotta Operations Center) connection dialog will prompt you for credentials before completing a server connection. The stop-tc-server

script

located

in

the

bin

directory

will

also

prompt

for

a

username

and

password.

h3. Configuration See the [Configuration Guide and Reference|Configuration Guide and Reference#server-authentication] for information on configuring the Terracotta server to use JMX authentication. h3. Permissions Setting permissions is done with the files *jmxremote.password* and *jmxremote.access.* Java security is very sensitive to the permission settings of these files so you will need to verify that they are properly set. One caveat to pay attention to is that the files must be owned by the system user who will be executing the Terracotta server. If the Terracotta server runs under the user "tcuser" then you must chown tcuser to change the ownership. The password file is most sensitive and requires: {noformat}

Configuration

See the Configuration Guide and Reference for information on configuring the Terracotta server to use JMX authentication.

Permissions

Setting permissions is done with the files jmxremote.password and jmxremote.access. Java security is very sensitive to the permission settings of these files so you will need to verify that they are properly set. One caveat to pay attention to is that the files must be owned by the system user who will be executing the Terracotta server. If the Terracotta server runs under the user "tcuser" then you must chown tcuser to change the ownership. The password file is most sensitive and requires:

No Format

# chmod 500 jmxremote.password 

-------------------------------------------------------------------------------
-rw-r--r-- 1 tcuser tcuser 2375 Feb 12 18:19 jmxremote.access
-r-x------ 1 tcuser tcuser 2873 Feb 12 18:19 jmxremote.password
-------------------------------------------------------------------------------
{noformat}

NOTE:

Setting

these

permissions

on

Windows

is

possible

but

requires

a

special

DOS

command.

This

permission

is

only

available

on

Windows

drives

with

NTFS

formatting.

See

http://java.sun.com/j2se/1.5.0/docs/guide/management/security-windows.html

for

more

information.

h3.

Users/Groups

You

may

append

to

the

default

files

located

in

$JAVA_HOME/jre/lib/management

using

the

following

commands:

{

No Format
}

# echo "myusername mypassword" >> jmxremote.password
# echo "myusername readwrite" >> jmxremote.access
{noformat}

'myusername'

defines

the

desired

username

and

'mypassword'

defines

the

desired

password.

In

jmxremote.access,

myusername

is

associated

with

the

username

defined

in

jmxremote.password

and

must

declare

readwrite

permissions

for

that

group.

{

Panel
:
title
=
Sample
excerpt
from
jmxremote.password
} {noformat}
No Format

...
# Following are two commented-out entries. The "measureRole" role has
# password "QED". The "controlRole" role has password "R&D".
#
# monitorRole QED
# controlRole R&D
myusername mypassword
...
{noformat} {panel} {panel:title=Sample excerpt from jmxremote.access} {noformat}
Panel
titleSample excerpt from jmxremote.access
No Format

...
# Default access control entries:
# o The "monitorRole" role has readonly access.
# o The "controlRole" role has readwrite access.

monitorRole readonly
controlRole readwrite
myusername readwrite
...
{noformat} {panel} {anchor:

Anchor
cluster-events
cluster-events

} h2. Cluster Events and Cluster Node Data The Terracotta Cluster Events JMX interface lets a Terracotta client listen for cluster events such as client connect and disconnect from the server and other clients connecting and disconnecting from the cluster. As of Terracotta 2.3, when a Terracotta client connects to the cluster, it publishes a local MBean to the server called "Terracotta Cluster Bean". You can register with that bean to listen to cluster events. The published events are: * *com.tc.cluster.event.thisNodeConnected*&mdash;The local client (the current JVM) connecting to a Terracotta server * *com.tc.cluster.event.thisNodeDisconnected*&mdash;The local client disconnecting from a Terracotta server * *com.tc.cluster.event.nodeConnected*&mdash;Another client connecting to a Terracotta server * *com.tc.cluster.event.nodeDisconnected*&mdash;Another client disconnecting to a Terracotta server You can also query the MBean for the local client's cluster node id (which is a unique id given to each client when it connects to the cluster for the first time) and for an enumeration of the node id's of each currently connected client. Using this data, you can keep a local model of the Terracotta cluster. You can also find out if the local client has been disconnected from the cluster and take any further action. Here's some sample code that will locate the local MBeanServer and register for cluster events: {code:Java} // get a reference to the local MBeanServer... final List servers = MBeanServerFactory.findMBeanServer(null); final MBeanServer server = (MBeanServer) servers.get(0); // Use these strings as the delegate name and cluster bean name final ObjectName delegateName = ObjectName.getInstance("JMImplementation:type=MBeanServerDelegate"); final ObjectName clusterBeanName = new ObjectName("org.terracotta:type=Terracotta Cluster,name=Terracotta Cluster Bean"); // Before you register a cluster bean listener, you have to wait for that cluster bean to be registered. // For an example of how to do that, see the ClusterBeanRegistrationListener in the tc-jmx-examples code waitForClusterBeanRegistration(); // Now that we're sure the cluster bean has been registered, we can add listeners. // // A cluster event listener should implement the interface: // // javax.management.NotificationListener // // // The callback method you implement is: // // public void handleNotification(Notification notification, Object handback) // // In that method, you can examine the type of the Notification object: // // notification.getType(); // // Check that type to see if it is one of the four cluster event types described above: // // com.tc.cluster.event.thisNodeConnected // com.tc.cluster.event.thisNodeDisconnected // com.tc.cluster.event.nodeConnected // com.tc.cluster.event.nodeDisconnected // server.addNotificationListener(clusterBeanName, myClusterEventListener, null, null); // Now that the cluster bean is registered, we can find out what our node id is System.err.println("My cluster node id is: " + server.getAttribute(clusterBeanName, "NodeId")); // and I can get a snapshot of all the connected clients (including myself). String[] clusterNodeIds = (String[]) server.getAttribute(clusterBeanName, "NodesInCluster"); for (int i=0; i<clusterNodeIds.length; i++) { System.err.println("Cluster node id: " + clusterNodeIds[i]); } {code} To see a sample usage of the Terracotta Cluster Events interface, check out the jmx-sample (http://svn.terracotta.org/svn/forge/projects/jmx-sample) project from the Terracotta forge. Then, start the Terracotta server. Next, run the *demo.ClusterEvents* class (in Eclipse as a Java Application or from the command line) with these VM arguments: {panel} -Xbootclasspath/p:/path/to/your/terracotta/installation/lib/dso-boot/dso-boot-hotspot_osx_150_07.jar -Dtc.install-root=/path/to/your/terracotta/installation -Dtc.config=localhost:9510 -Dcom.sun.management.jmxremote {panel} *Note:* * Change _/path/to/your/terracotta/installation_ to wherever you installed Terracotta * Change _dso-boot-hotspot_osx_150_07.jar_ to the appropriate boot jar for your platform * You can also run _dso-env.sh_ or _dso-env.bat_ to determine the proper values for these properties * The property _-Dcom.sun.management.jmxremote_ lets you use _jconsole_ to inspect the published MBeans. You should see something like this: {panel} 2007-04-03 14:49:59,929 INFO - Terracotta version 2.3.0-stable0, as of 20070320-190325 (Revision 1952 by cruise@rh4mo0 from 2.3) 2007-04-03 14:50:00,484 INFO - Configuration loaded from the server at 'localhost:9510'. 2007-04-03 14:50:00,601 INFO - Log file: '/Users/orion/Documents/workspace/tc-jmx-samples/logs/client-logs/terracotta-client.log'. Creating a new ClusterEvents object and calling run()... My cluster node id is: 11 Cluster node id: 11 Now I'm going to wait forever... {panel} You can then start another Terracotta client (one of the samples that comes with the Terracotta kit will work) and the *demo.ClusterEvents* program will print the client connect event to the Eclipse console. If you quit that client, the *demo.ClusterEvents* program will print the client disconnect event to the console: {panel} Received a cluster event notification: javax.management.Notification\[source=org.terracotta:type=Terracotta Cluster,name=Terracotta Cluster Bean\]\[type=com.tc.cluster.event.nodeConnected\]\[message=12\] Received a cluster event notification: javax.management.Notification\[source=org.terracotta:type=Terracotta Cluster,name=Terracotta Cluster Bean\]\[type=com.tc.cluster.event.nodeDisconnected\]\[message=12\] {panel} If you stop the Terracotta server, the *demo.ClusterEvents* program will print a notification that this node has been disconnected: {panel} Received a cluster event notification: javax.management.Notification\[source=org.terracotta:type=Terracotta Cluster,name=Terracotta Cluster Bean\]\[type=com.tc.cluster.event.thisNodeDisconnected\]\[message=11\] {panel} If the Terracotta server is configured to be persistent, you can start the server up again and the *demo.ClusterEvents* program will print a notification that this node has been connected: {panel} Received a cluster event notification: javax.management.Notification\[source=org.terracotta:type=Terracotta Cluster,name=Terracotta Cluster Bean\]\[type=com.tc.cluster.event.thisNodeConnected\]\[message=11\] {panel} For another example of the Terracotta Cluster Events interface in use, see the *chatter* sample in the Terracotta kit. *Further reading:* For more information on the Terracotta cluster architecture, see the [Concept and Architecture Guide|Concept and Architecture Guide#architecture] {anchor:dgc} h2. Distributed Garbage Collection *TODO:* Write me. http://jira.terracotta.org/jira/browse/DOC-139 {anchor:rogueclientdetection} h2. Rogue/Slow/Unhealthy Client Detection and monitoring Rogue Client is a client which is for some reason not healthy e.g its having a lot of pending transactions or its slow.Terracotta server exposes a list of MBeans to monitor the performance of these clients (like pending transactions) so that one can determine the health of a client and can kill any particular unhealthy client. Here is some sample code for getting client MBeans {code:Java} MBeanServerConnection mbsc = null; JMXConnectorProxy jmxc = new JMXConnectorProxy("localhost", Integer.valueOf(appConfig.getAttribute(JMX_PORT))); try { mbsc = jmxc.getMBeanServerConnection(); } catch (IOException e) { throw new AssertionError(e); } DSOMBean dsoMBean = (DSOMBean) MBeanServerInvocationHandler .newProxyInstance(mbsc, L2MBeanNames.DSO, DSOMBean.class, false); ObjectName[] clientObjectNames = dsoMBean.getClients(); DSOClientMBean[] clients = new DSOClientMBean[clientObjectNames.length]; for (int i = 0; i < clients.length; i++) { clients[i] = (DSOClientMBean) MBeanServerInvocationHandler.newProxyInstance(mbsc, clientObjectNames[i],DSOClientMBean.class, false); } {code} These are the list of MBeans exposed to monitor these clients. The name of the APIs are self explanatory * *com.tc.stats.isTunneledBeansRegistered*&mdash;Is the bean registered * *com.tc.stats.getL1InfoBeanName*&mdash;Get the name(class ObjectName) of this bean * *com.tc.stats.getL1InfoBean*&mdash;Get the MBean * *com.tc.stats.getInstrumentationLoggingBeanName*&mdash;Get the name(class ObjectName) of the instrumentation logging MBean) * *com.tc.stats.getInstrumentationLoggingBean*&mdash;Get the Instrumentation logging MBean * *com.tc.stats.getRuntimeLoggingBeanName*&mdash;Get the name(Class ObjectName) of te runtime logging MBean * *com.tc.stats.getRuntimeLoggingBean*&mdash;Get the run time logging MBean * *com.tc.stats.getRuntimeOutputOptionsBeanName*&mdash;Get the name(class ObjectName) of the runtime output options MBean * *com.tc.stats.getRuntimeOutputOptionsBean*&mdash;Get the runtime output options MBean * *com.tc.stats.getChannelID*&mdash;Get the channel ID * *com.tc.stats.getRemoteAddress*&mdash;Get the remote address of the channel * *com.tc.stats.getTransactionRate*&mdash;Get the transaction rate for the client * *com.tc.stats.getObjectFaultRate*&mdash;Get the object fault rate for the client * *com.tc.stats.getObjectFlushRate*&mdash;Get the object flush rate for the client * *com.tc.stats.getPendingTransactionsCount*&mdash;Get the pending transactions count for the client * *com.tc.stats.getStatistics*&mdash;Get the statistics for the given properties * *com.tc.stats.killClient*&mdash;Kill this client ---- *TODO:* Document the rest of the MBeans. {div9}

Platform Cluster Events and Cluster Node Data

A platform cluster-events API that allows events to be communicated using a standard Java programming approach is available as part of the Terracotta API introduced with Terracotta 3.0.0. This Java-based events API can be more easily integrated into a Java applications environment.

Previous versions of Terracotta relied on JMX-based cluster events. This has caused issues for applications attempting to integrate with Terracotta because most Java applications do not use JMX in this way. Beginning with Terracotta 3.0.0, these JMX-based cluster events are no longer supported.

Applications that rely on the obsoleted JMX API for cluster events must be rewritten to use the new Java API. Support for the JMX cluster events will be removed without further notice. Application code using the TerracottaCluster mbean or registering notifications with it will receive an UnsupportedOperationException.

comment

{anchor:dgc}
h2. Distributed Garbage Collection
*TODO:* Write me.
http://jira.terracotta.org/jira/browse/DOC-139

Anchor
rogueclientdetection
rogueclientdetection

Rogue/Slow/Unhealthy Client Detection and monitoring

A rogue client is a client which is for some reason not healthy e.g its having a lot of pending transactions or its slow.Terracotta server exposes a list of MBeans to monitor the performance of these clients (like pending transactions) so that one can determine the health of a client and can kill any particular unhealthy client.

Here is some sample code for getting client MBeans

Code Block
Java
Java

MBeanServerConnection mbsc = null;
JMXConnectorProxy jmxc = new JMXConnectorProxy("localhost", Integer.valueOf(appConfig.getAttribute(JMX_PORT)));
try {
  mbsc = jmxc.getMBeanServerConnection();
} catch (IOException e) {
  throw new AssertionError(e);
}
DSOMBean dsoMBean = (DSOMBean) MBeanServerInvocationHandler
          .newProxyInstance(mbsc, L2MBeanNames.DSO, DSOMBean.class, false);

ObjectName[] clientObjectNames = dsoMBean.getClients();
DSOClientMBean[] clients = new DSOClientMBean[clientObjectNames.length];
for (int i = 0; i < clients.length; i++) {
  clients[i] = (DSOClientMBean) MBeanServerInvocationHandler.newProxyInstance(mbsc, clientObjectNames[i],DSOClientMBean.class, false);
}

This is a partial list of MBeans exposed to monitor these clients:

  • com.tc.stats.isTunneledBeansRegistered—Is the bean registered
  • com.tc.stats.getL1InfoBeanName—Get the name(class ObjectName) of this bean
  • com.tc.stats.getL1InfoBean—Get the MBean
  • com.tc.stats.getInstrumentationLoggingBeanName—Get the name(class ObjectName) of the instrumentation logging MBean)
  • com.tc.stats.getInstrumentationLoggingBean—Get the Instrumentation logging MBean
  • com.tc.stats.getRuntimeLoggingBeanName—Get the name(Class ObjectName) of te runtime logging MBean
  • com.tc.stats.getRuntimeLoggingBean—Get the run time logging MBean
  • com.tc.stats.getRuntimeOutputOptionsBeanName—Get the name(class ObjectName) of the runtime output options MBean
  • com.tc.stats.getRuntimeOutputOptionsBean—Get the runtime output options MBean
  • com.tc.stats.getChannelID—Get the channel ID
  • com.tc.stats.getRemoteAddress—Get the remote address of the channel
  • com.tc.stats.getTransactionRate—Get the transaction rate for the client
  • com.tc.stats.getObjectFaultRate—Get the object fault rate for the client
  • com.tc.stats.getObjectFlushRate—Get the object flush rate for the client
  • com.tc.stats.getPendingTransactionsCount—Get the pending transactions count for the client
  • com.tc.stats.getStatistics—Get the statistics for the given properties
  • com.tc.stats.killClient—Kill this client
  • *com.tc.stats.

Hibernate Statistics From Terracotta Clients

If you are running Terracotta clustered caches as hibernate second-level cache provider, it is possible to access the hibernate statistics + ehcache stats etc via jmx. EhcacheHibernateMBean is the main interface that exposes all the API's via jmx. It basically extends two interfaces – EhcacheStats and HibernateStats. And as the name implies EhcacheStats contains methods related with Ehcache and HibernateStats related with Hibernate. You can see cache hit/miss/put rates, change config element values dynamically – like maxElementInMemory, TTI, TTL, enable/disable statistics collection etc and various other things. Please look into the specific interface for more details.