- How DSO Clustering Works
- Platform Concepts
- Hello Clustered World
- Setup and Configuration
- Planning for a Clustered App
- Configuring Terracotta DSO
- Configuration Reference
- Using Annotations
- Cluster Events
- Data Locality Methods
- Distributed Cache
- Clustered Async Data Processing
- Tool Guides
- Developer Console
- Operations Center
- tim-get (TIM Management Tool)
- Platform Statistics Recorder
- Eclipse Plugin
- Sessions Configurator
- Clustering Spring Webapp with Sessions Configurator
- Testing, Tuning, and Deployment
- Top 5 Tuning Tips
- Testing a Clustered App
- Tuning a Clustered App
- Deployment Guide
- Operations Guide
- FAQs and Troubleshooting
- General FAQ
- DSO Technical FAQ
- Troubleshooting Guide
- Non-portable Classes
- Migrating From DSO
- Concept and Architecture Guide
- Examinator Reference Application
- Clustered Data Structures Guide
- Integrating Terracotta DSO
- Clustering Spring Framework
- Integration Modules Manual
- AspectWerkz Pattern Language
Publish Date: November, 2011
Configuring Terracotta DSO
This guide contains instructions on how to set up your
tc-config.xml file so that your application will be clustered with Terracotta Distributed Shared Objects (DSO).
- Overview of the Configuration Process
- Identifying a root
- A basic configuration file
- Configure a root
- Starting a Terracotta Server Instance
- Start your application
- Instrumentation and Locking
- Resolving TCNonPortableObjectError
- Resolving UnlockedSharedObjectException - Configuring Locking
- Confirm that state replication was successful
Overview of the Configuration Process
Getting Terracotta integrated is a simple process. Using a configuration file (or annotations), you tell Terracotta which classes need to be instrumented and what locking should be provided to replicate your shared state across many JVMs.
The basic steps of this process are:
- (optional) Identify a root
- Create your config file
- Start a Terracotta Server Instance
- Start your Application (a Terracotta Client)
- (iteratively) resolve UnlockedSharedObject and TCNonPortableErrors
The process is approximated by the diagram to the right.
Identifying a root
All Terracotta applications have a root. A root is a core concept in Terracotta. Roots are the top of a clustered graph. They are typically a collection such as a HashMap or ArrayList. They can also be a POJO.
You can read more about roots in the Concept and Architecture Guide.
You may not have to specify a root, depending on your application. If you are using EHCache, and are using the EHCache TIM, then the root selection is automatically made for you inside the EHCache TIM (what this means is that the Integration Module for EHCache already knows how to cluster EHCache instances and does not need an additional definition of a root to function properly).
A basic configuration file
Next you will need a basic configuration file (if you do not already have one). Here is the most basic configuration file (if you already have a configuration file, use that one instead):
Save this file to a file named
tc-config.xml. You must now update your config file to contain a root.
Configure a root
To configure a root, first identify a point in your application that represents shared state. Usually this shared state is held by a HashMap, an Array, or some other java.util Collection.
For example, in the HashMap Recipe the HashMap held by the field
map is configured to be a root.
The section in the
tc-config.xml file that controls roots is called
roots and should reside inside the
application/dso xml element.
The HashMap Recipe config is repeated here for reference. Notice the
roots section, which identifies the
map field to be a root in the Main class:
Starting a Terracotta Server Instance
By default, server instances use the file
tc-config.xml, if one exists in the local directory. For this example, be sure the configuration file you edited is saved in $TC_HOME before starting a Terracotta server instance.
To start a Terracotta server instance:
If the server instance started successfully, you should see output similar to the following:
Note the line that states what configuration file was used to start the server. This configuration file should correspond to the one you created.
Start your application
Now that you have a running server instance and a working configuration file, you can start your application. Because we have configured only a root, and not instrumentation or locking, you will encounter exceptions thrown by Terracotta. These exceptions can assist you in configuring your application correctly.
Instrumentation and Locking
During the integration process, it is common to receive the following types of exceptions from Terracotta:
- TCNonPortableObjectError exception
- UnlockedSharedObjectException exception
Don't panic. These are normal!
Next, we will describe how to resolve them.
This error is caused by an attempt to share an object that cannot be shared, or a non-portable object. More specifically, the error has one of the following conditions:
- An attempt was made to share a class that has not been instrumented by Terracotta.
- An attempt was made to share a non-portable class that has been instrumented by Terracotta.
To see a solution diagram of how this error is handled, click the following graphic.
To resolve the second case, a non-portable instrumentation error, either mark the section of the graph as transient, or refactor your application. The rest of this section deals with the first case, where a portable class that was not instrumented is being shared.
When you receive TCNonPortableObjectError, Terracotta gives you suggestions of how to resolve it. For example, let's look at the Instrumentation Recipe, which includes a Terracotta configuration file that instruments a class called Counter. For this example, we ignore the locking section and remove the section that instruments Counter, leaving the following configuration:
Now when we run this application:
we are rewarded with a NonPortableObjectError that looks like this:
Notice at the end of the exception, it tells us how to fix our config file. In general, there are three ways to deal with TCNonPortableObjectError:
- Add Instrumentation: update your tc-config.xml with the appropriate configuration
- Mark Transient: mark the offending object (or object graph) as transient in your tc-config.xml
- Refactor: refactor your code to avoid putting the offending instance into the clustered graph
If the error message instructs you to add a class to the Terracotta boot JAR, see the boot jar section in the Troubleshooting Guide.
We need to add the
Counter class to the instrumentation list. This is done by adding the snippet of configuration provided in the Exception to the
application/dso XML element. Our resulting
tc-config.xml file now looks like this:
In your application, you may have to repeat this process several times until all of the classes that are being shared are accounted for.
Once you have updated your configuration file, restart your application again and continue to resolve any further Exceptions.
Certain classes causing the TCNonPortableObjectError exception are truly non-portable and cannot be instrumented. Terracotta can be configured to avoid trying to share these classes using transience. Terracotta can recognize transient classes if they are declared transient in code or in
Be sure to mark the offending class highest in the entire object graph that's being added. Marking the immediate offending class may not prevent the TCNonPortableObjectError exception.
For more information on transience, see Transience in Terracotta in the DSO Concept and Architecture Guide.
Once you have updated your configuration file, restart your application again and continue to resolve any further Exceptions.
If instrumenting or marking as transient are insufficient for eliminating the problem you are running into, consider refactoring your code. Code that was not written to cleanly separate shared state from non-shared state may be too problematic to fix using the instrumentation and transient strategies.
More Information on Portability
- The non-portable-dump section of the Configuration Guide and Reference shows how to analyze the log when a TCNonPortableError occurs.
- The DSO Concept and Architecture Guide explains portability in Terracotta.
- The DSO Troubleshooting Guide has example code and a discussion of how TCNonPortableError could be generated and resolved.
Resolving UnlockedSharedObjectException - Configuring Locking
An UnlockedSharedException simply means that Terracotta has detected that our application has tried to update a value in a shared object without a Terracotta lock present.
Terracotta locks are very much the same as standard Java synchronization. The big difference between Terracotta locking and Java synchronization is that Terracotta requires that you have a lock to update a shared object, while Java will let this happen without complaining.
This is actually a good thing, because with Terracotta, your application is now running in a clustered environment, and a clustered environment is equivalent to having multiple threads. When multiple threads have write access to shared data, just as in normal Java, you must always protect access to that data. Terracotta saves you from finding out race conditions or data corruption at run-time by detecting and reporting attempts to update shared data outside of a lock.
For more details on the similarities, and differences, between Terracotta locking and Java locking, read the DSO Concept and Architecture Guide Locks Section.
Let's use the Instrumentation Recipe again. So far, we have added the proper instrumentation configuration to our config file. However, when we run this example again, it tries to update a counter in a shared object, so we now get an UnlockedSharedException, which looks like this:
This tells us several things. First of all, we know that our application tried to update a field on a shared object without a Terracotta lock present.
This can be due to one or more of the following:
- We did not configure Terracotta locking for this code.
- The code itself does not have synchronization that Terracotta can use as a boundary.
- The class doing the locking must be included for instrumentation.
- The object was first locked, then shared (for an example, see this gotcha).
- This error can also be caused by not specifying the fully qualified method name in a lock expression.
Here's how we can resolve these issues. First, identify the method upon which the lock should be applied.
Identify the method for configuration
We can identify what method was in question by analyzing the stack trace. Let's look at the stack-trace:
In this stack trace, the method that is causing the problem is
Counter.count(). We can figure this out by looking for the line directly below the first line of Terracotta injected code. Terracotta inject code always begins with
_tc so the first line in this stack trace of Terracotta injected code is at Counter.__tc_setcount(Counter.java)}}.
The line directly below this code is
Counter.count(Counter.java:13) meaning the
Counter.count() method is to blame for this particular exception. Furthermore, we can actually make a pretty good guess as to what went wrong - first of all, we can see that the exact spot in the code that tripped us up was line 13. If you have the source code, pull up the Counter class and you can see what the code is that triggered this exception.
Here it is, from the Counter.java class in the Instrumentation recipe:
Even if we don't have access to the source, we can still make a pretty good guess what happened. This is because the Terracotta injected code gives us a clue. The injected code for a field setter will always take the form of
fieldname is the name of the field, so we know from this particular code that the
Counter class was trying to update the
count field when the exception was thrown.
So, now we know what method is causing our problem. Depending on the method, there may or may not be synchronization appropriate for auto-locking. If there is, proceed to the next section. If there is not, proceed to the #Adding Terracotta locking for code without synchronization section.
Adding Terracotta auto-locking for code with synchronization
If your code already has synchronization, then the only thing you have to do is add the proper Terracotta configuration to auto-lock the code.
Auto-locking is a simple process - it tells Terracotta that for any instance of synchronization it finds within the method you configure, add a Terracotta lock boundary to that code. In effect, this converts a single JVM synchronized boundary into a cluster-wide lock boundary.
In our example above, we can see that in fact the Counter class already has synchronization on it, so we just need to add the correct Terracotta configuration to make this example work.
We add the auto-lock to the
application/dso/locks section (if there is not a locks section, you will need to add it). We must specify the method-expression field, and a lock-level. The lock-level defaults to write, so you can leave that off if you like. When we are done, our config file will look like this:
Adding Terracotta locking for code without synchronization
If no synchronization exists in the method, you can add a Terracotta lock in one of two ways:
- Use auto-synchronized
- Use named locks
Adding locking using auto-synchronized
Auto-synchronized has the same effect as adding the synchronized keyword to the method before adding Terracotta locking. You can imagine as your class is loaded, Terracotta instruments the class and adds synchronization to the method you have specified as auto-synchronized. Then it adds the auto-lock.
To auto-synchronize a method, use the
auto-synchronized="true" attribute on the auto-lock. Adding auto-synchronized, instead of just an auto-lock, to the Counter example, the
tc-config.xml file would look like this:
Adding locking using named locks
If none of the above methods are sufficient for your needs, you can resort to adding a named lock. You should avoid named locks unless absolutely necessary as named locks are global, meaning the lock is very coarse-grained, and may impact the performance of your application.
Adding named locks is very similar to adding auto-synchronized, however in the auto-synchronized case, the lock is acquired on the shared object, whereas a named lock is acquired globally for all methods locked with that name.
To add a named lock, use the named lock syntax, like so:
Confirm that state replication was successful
Once you are finished updating your configuration file, and your application appears to run correctly with Terracotta installed, you can now test to see if the state is truly replicated.
Start a Console Session
Launch the Terracotta Developer Console and inspect the data in the object browser. This is a good way to use a single JVM process to verify that the data you expect to be replicated is in fact shared with Terracotta.
Start a second JVM
Once your application is working with a single JVM, you must check it with 2 (or more) JVMs. Start your application again. Of course, depending on your application, you may or may not be able to do this without changes to your application code.
For example, you may need to assign different responsibilities to each JVM. It may be that the first JVM may need to be a Producer, and the second a Consumer. Or it may be that all JVMs are equivalent (e.g. a Web Application processing User Requests). Depending on your application, you may need to factor each Main process into one or more differing startup modes.