Session Replication in Tomcat 5 Clusters, Part 2

This is the second part of a series on session replication in the Tomcat 5 container. In part one, I provided an overview of sticky versus replicated sessions, and session replication design considerations. I also discussed how session replication works in a Tomcat cluster, especially when a server is started up or shut down. In this part, I will cover the configuration details of a sample Tomcat cluster setup and compare different session replication scenarios and their respective impacts on cluster performance.

Cluster Setup

To enable session replication in the Tomcat 5 container, the following steps should be completed:

• For clustering to work, you need to enable multicasting on your system.
• To use session replication efficiently, all Tomcat instances should be configured the same. This means the web applications should be deployed homogeneously to every server instance in the cluster. This configuration also simplifies cluster administration, maintenance, and troubleshooting tasks.
• Uncomment Cluster and Valve (ReplicationValve) elements in server.xml. Enabling the Cluster element in server.xml means that the session manager for all the web contexts will be changed. So when running a cluster, make sure that you only have the web apps that need to be clustered and remove the other ones.
• If the server instances are running on the same machine, make sure the tcpListenPort attribute is unique for each instance.
• Make sure web.xml has the distributable element.
• All the session attributes must implement the java.io.Serializable interface.

The sample cluster setup has two Tomcat instances and a load balancer to distribute the requests between the servers. All three servers were launched on a single machine. The following table lists the configuration parameters of the cluster and load balancer.

Editor's note: the values for Web App Directory above have linefeeds inserted to suit this article's page layout. In your deployment, each should be a single, unbroken line.

Related Reading Tomcat: The Definitive Guide By Jason Brittain, Ian F. Darwin

Note: Do not set 127.0.0.1 as the tcpListenAddress, since 127 is used as the loopback address and may cause problems during session replication.

Load Balancer Setup

Tomcat server doesn't provide the failover capability to redirect any incoming requests to the next available server in case one of the cluster nodes goes down. So I used a separate load balancer to take care of load balancing the web requests. Some popular load balancers are Apache/mod_jk, Balance, and Pen. I used Pen as the load balancer in the sample clustering setup.

Pen is a simple load balancer for TCP-based protocols such as HTTP or SMTP. It allows several servers to appear as one to the outside, and automatically detects servers that are down and distributes clients across the available servers. It provides both load balancing and failover capabilities. Pen is easy to install and configure, is very simple to use, and works on Windows and UNIX operating systems. The sample configuration uses the round-robin load balancing algorithm for distributing the load between the cluster nodes.

The command to run the load balancer is as follows:
pen -r -a -f -d localhost:8080 192.168.0.10:9080 192.168.0.20:10080

The following is the explanation of each command-line parameter used to launch the load balancer.

-r: Use round robin for load balancing
-a: Prints the data sent back and forth in ASCII format
-f: Stay in foreground
-d: Enable debug mode

For more details on the other parameters used to launch Pen, refer to the Pen web site.

Figure 1 shows the topology diagram with two cluster nodes, the load balancer, the instrumentation layer, and the test client.

Figure 1. Cluster setup diagram (Click on the screen shot to open a full-size view)

Test Setup

I created a web application called clusterapp to demonstrate the clustering setup and the session replication mechanisms. To test the actual session replication, I wrote a sample Java client called SessionReplicationClient to measure how long it took to copy the session data from one server to the other. This client uses Jakarta Commons HttpClient framework to create and manipulate the HTTP session and calls a servlet on the Tomcat server. The hardware/software specifications of the machine used to test the session replication are listed below.

• CPU: HP Pavilion Pentium III 800MHz
• Memory: 512MB RAM
• Hard disk: 40GB
• Operating system: Windows 2000 server
• JDK version: 1.4.2_05 (Note: JDK 1.4 or later version is required to use clustering and session replication)
• Tomcat version: 5.0.28
• Tools: Pen, Log4J, Eclipse, Commons HttpClient

When the replication client program is run, it first sets an action command on how to handle the session (i.e. add a new attribute to the session, remove an existing attribute from the session, or invalidate the session). The client then calls the ReplicationServlet by using the HttpClient and PostMethod classes in the Commons HttpClient framework. Based on the session command, the servlet either generates a new session or modifies the existing session and forwards to a web page to display the session details. If there is an error in managing the session, it forwards to an error page. I wrote a custom session listener class (ClusterAppSessionListener) to track the session management details as new sessions are created and the existing ones are altered or expired.

The flow of the session replication is represented in the sequence diagram in Figure 2.

Figure 2. Session replication sequence diagram (Click on the screen shot to open a full-size view)

The session state on the server is tracked by a cookie for each web request, so the request URL must be the same from the client side to keep using the same session. Otherwise, a new HTTP session will be created every time a request is made. I tested the replication with two types of tests based on the type of attributes added to the session. The first category had 100 lightweight objects (1K each) added to the session. In the second category, I added a single large object (100K) to compare the session replication time taken based on the session attribute size.

Listed below are the test case specifications of SessionReplicationClient:

• Number of client threads: 2
• Number of repetitions: 1000
• Delay between requests: 1000 milliseconds (2000ms for large sessions)
• Number of test samples: 1000
• Session Attributes: Small (100 1K objects) or large (one 100K object)

The command used to run the test client with the required arguments is as follows.

The test scenarios included using different session managers (SimpleTcpReplicationManager or DeltaManager) and session replication modes (Async, Sync, or Pooled). The following table shows the list of scenarios tested in the Tomcat cluster.

To change replication mode from pooled to synchronous to asynchronous, just change the replicationMode attribute value in the Sender tag in the server.xml file. Also, to change the session manager type, change the managerClassName attribute in the Cluster element.

The following parameters were used to compare the response times and effectiveness of session replication.

• Average response time (ms)
• Average request time (ms)
• Cluster overhead time (ms)
• Replication time (ms)
• Ratio (bytes/ms)
• Ratio (bytes/request)

Test Results

The combination of using delta manager and pooled replication mode was the best criteria for efficient session replication. Also, keeping the session size small proved to be two to three times faster than replicating large sessions. The results from the session replication tests are listed in the table below.

Replication Manager

DeltaManager proved to be more efficient in replicating sessions because it only deals with session deltas, not the entire session's data. Session replications using DeltaManager were 30 (for large sessions) to 50 (small sessions) percent more efficient, compared to using the simple replication manager.

Mode of Replication

Pooled replication mode took less time to replicate the sessions compared to the other two options (Synchronous or Asynchronous). The pooled option was almost four times as fast as the async option, in terms of replication time. But the response time and cluster overhead times for pooled and async modes were almost same because in the asynchronous mode, Cluster doesn't wait for the session to be completely replicated before returning.

Conclusion

Based on the results from session replication tests we can conclude that DeltaManager should be used whenever possible. Since the overhead in replicating session data is significant, make sure that you don't store too much data in the session. Also, the size of the attributes added to the session is another factor that impacts the session replication time. When I ran the test to store big objects (100K) in the session, the replication times were very high compared to the test with small objects (1K) in the session. The other best practices to minimize the session replication overhead are to avoid calling session.setAttribute() and to store data in the request object instead of the session. This is a better option because the request attributes are reset when the web request is completed. Also, if there is no business reason to store the data on the server, we can store it on the client in the form of cookies. This approach completely eliminates the need for using any session at all.

One way to minimize the time taken for session replication is to limit the number of server instances in the cluster to two (or maybe three) servers so that session data on the first server is copied to the second server in case the first one goes down. In order to keep the network traffic down in an all-to-all environment, the cluster can be split into smaller groups, with each group consisting of two or three server instances. Remember, the more server instances in the cluster, the longer the session replication time will be. Tomcat 5 currently doesn't support this concept of primary/secondary replication. It will be available in a future release, where we will be able to store the session on one or two backup servers. With this feature, Tomcat will provide a more complete session replication solution for running web applications in cluster environments.

Some of the other features that Tomcat will support in the future are:

• The ability to have attributes that are non-serializable in the session, which the cluster will ignore.
• The ability to replicate context attributes and non-session related data, such as JDBC connection pools and object caches.

These new features will make the session replication in Tomcat clusters more robust and flexible.

I want to thank Filip Hanik for his valuable suggestions and feedback on the replication modes, session replication web application setup, and the test client sections of this article.

Resources

• Session Replication sample code for this article
• Tomcat Clustering/Session Replication Home Page
• Tomcat Load Balancer Home Page
• "Clustering and Load Balancing in Tomcat 5, Part 1"
• "Clustering and Load Balancing in Tomcat 5, Part 2"
• "BEA WebLogic Server: Using WebLogic Server Clusters"
• Patterns of Enterprise Application Architecture

Srini Penchikala is an information systems subject matter expert at Flagstar Bank.

Return to ONJava.com

Copyright © 2009 O'Reilly Media, Inc.