Clustering Tomcat

I’ve been meaning for some time to set up Tomcat clustering in several different ways just to see how it works. I finally got a chance today. There are guides and other bits of information elsewhere on how to do all this. I’m not gonna tell you how, sorry; the point of this post is to document my problems and how I overcame them.

A word about setup

My front end load balancer is Apache httpd 2.2.15 (from SpringSource ERS 4.0.2) using mod_proxy_http, mod_proxy_ajp, mod_jk, and (just to keep things interesting) mod_rewrite as connectors to the backend Tomcat 6.0.28 instances. I wanted to try out 2-node Tomcat clusters (one side of each cluster ERS Tomcat, the other side tc Server 2.0.3) without any session replication (so, sticky sessions and you get a new session if there’s a failure) and with session replication via the standard Delta manager (which replicates all sessions to all nodes) and the Backup manager (which replicates all sessions to a single reference node, the “backup” for each app).  Basically the idea is to test out every conceivable way to put together a cluster with standard SpringSource technologies.

The first trick was mapping all of these into the  URL space for my httpd proxy. I wanted to put each setup behind its own URL /<connector>/<cluster> so e.g /http/backup and /ajp/delta. This is typically not done and for good reason; mod_proxy will let you do the mapping and will even clean up redirects and cookie paths from the backend, but to take care of self-referential links in the backend apps you actually have to rewrite the content that comes back; for that I installed mod_proxy_html, a non-standard module for doing such things. The reason it’s a non-standard module is that this approach is fraught with danger. But given that I mostly don’t care about how well the apps work in my demo cluster, I thought it’d be a great time to put it through its paces.

For this reason, most people make sure the URLs on the front-end and back-end are the same; and in fact, as far as I could tell, there was no way at all to make mod_jk do any mapping, so I’m setting it up as a special case – more later if it’s relevant. The best way to do this demo would probably be to set up virtual hosts on the proxy for each scenario and not require any URI mapping; if I run into enough problems I’ll probably fall back to that.

Problem 1: the AJP connector

I got things running without session replication fairly easily. My first big error with session replication was actually something else, but at first I thought it might be related to this warning in the Tomcat log:

Aug 13, 2010 9:43:00 PM org.apache.catalina.core.AprLifecycleListener init

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/jdk1.6.0_21/jre/lib/i386/server:/usr/java/jdk1.6.0_21/jre/lib/i386:/usr/java/jdk1.6.0_21/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
Aug 13, 2010 9:43:00 PM org.apache.catalina.startup.ConnectorCreateRule _setExecutor
WARNING: Connector [org.apache.catalina.connector.Connector@1d6f122] does not support external executors. Method setExecutor(java.util.concurrent.Executor) not found.

These actually turn out to be related:

<Listener className=”org.apache.catalina.core.AprLifecycleListener” SSLEngine=”on” />

…

<Connector  executor=”tomcatThreadPool”
port=”8209″
protocol=”AJP/1.3″
emptySessionPath=”true”
/>

I don’t really know why the APR isn’t working properly, but a little searching turned up some obscure facts: if the APR isn’t loaded, then for the AJP connector Tomcat makes a default choice of implementation that doesn’t use the executor thread pool. So you have to explicitly set the class to use like this:

<Connector  executor=”tomcatThreadPool”
port=”8209″
protocol=”org.apache.coyote.ajp.AjpProtocol”
emptySessionPath=”true”
/>

Nice, eh? OK. But that was just an annoyance.

Problem 2: MBeans blowup

The real thing holding me back was this error:

Aug 13, 2010 9:52:16 PM org.apache.catalina.mbeans.ServerLifecycleListener createMBeans
SEVERE: createMBeans: Throwable
java.lang.NullPointerException
at org.apache.catalina.mbeans.MBeanUtils.createObjectName(MBeanUtils.java:1086)
at org.apache.catalina.mbeans.MBeanUtils.createMBean(MBeanUtils.java:504)

…

Now what the heck was that all about? Well, I found no love on Google. But I did eventually guess what the problem was. This probably underscores a good methodology: when working on configs, add one thing at a time and test it out before going on. If only Tomcat had a “configtest” like httpd – it takes forever to “try it out”.

In case anyone else runs into this, I’ll tell you what it was. The Cluster Howto made it pretty clear that you need to set your webapps context to be distributable for clustering to work. It wasn’t clear to me where to put that, but I didn’t want to create a context descriptor for each app on each instance. I knew you could put a <Context> element in server.xml so that’s right where I put it, right inside the <Host> element:

<Context distributable=”true” />

Well, that turns out to be a bad idea. It causes the error above. So don’t do that. For the products I’m using, there’s a single context.xml that applies to all apps on the server; that’s where you want to put the distributable attribute.

Cluster membership – static

My next task was to get the cluster members in each case to recognize each other and replicate sessions. Although all the examples use multicast to do this, I wanted to try setting up static membership, because I didn’t want to look up how to enable multicast just yet. And it should be simpler, right? Well, I had a heck of a time finding this, but it looks like the StaticMembershipInterceptor is the path.

My interceptor looks like this:

<Interceptor className=”org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor”>
<Member className=”org.apache.catalina.tribes.membership.StaticMember”
port=”8210″ securePort=”-1″
host=”vm-centos-cluster-tcs.sosiouxme.lan”
domain=”delta-cluster”
uniqueId=”{0,1,2,3,4,5,6,7,8,9}”
/>
</Interceptor>

(with the “host” being the other host in the 2-node cluster on each side). Starting with this configuration brings an annoying warning message:

WARNING: [SetPropertiesRule]{Server/Service/Engine/Cluster/Channel/Interceptor/Member} Setting property ‘uniqueId’ to ‘{1,2,3,4,5,6,7,8,9,0}’ did not find a matching property.

So I guess that property has been removed and the docs not updated; didn’t seem necessary anyway given the host/port combo should be unique.

In any case, the log at first looks encouraging:

Aug 13, 2010 10:13:04 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://vm-centos-cluster-tcs.sosiouxme.lan:8210,vm-centos-cluster-tcs.sosiouxme.lan,8210, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={100 101 108 116 97 45 99 108 117 …(13)}, ]
Aug 13, 2010 10:13:04 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:4
Aug 13, 2010 10:13:04 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{172, 31, 1, 108}:8210,{172, 31, 1, 108},8210, alive=25488670,id={66 108 -53 22 -38 -64 76 -110 -110 -54 28 -11 -126 -44 66 28 }, payload={}, command={}, domain={}, ]

Sounds like the other cluster member is found, right? I don’t know why it’s in there twice (once w/ hostname, once with IP) but I think the first is the configuration value, and the second is for when the member is actually contacted (alive=).

And indeed, for one node, later in the log as the applications are being deployed, I see the session replication appears to happen for each:

WARNING: Manager [localhost#/petclinic], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://vm-centos-cluster-tcs.sosiouxme.lan:8210,vm-centos-cluster-tcs.sosiouxme.lan,8210, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={100 101 108 116 97 45 99 108 117 …(13)}, ]. This operation will timeout if no session state has been received within 60 seconds.
Aug 13, 2010 10:13:09 PM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions
INFO: Manager [localhost#/petclinic]; session state send at 8/13/10 10:13 PM received in 186 ms.

Um, why is that a WARNING? Looks like normal operation to me, surely it should be an INFO. Whatever. The real problem is on the other side of the node:

13-Aug-2010 22:25:16.624 INFO org.apache.catalina.ha.session.DeltaManager.start Register manager /manager to cluster element Engine with name Catalina
13-Aug-2010 22:25:16.624 INFO org.apache.catalina.ha.session.DeltaManager.start Starting clustering manager at/manager
13-Aug-2010 22:25:16.627 WARNING org.apache.catalina.ha.session.DeltaManager.getAllClusterSessions Manager [localhost#/manager], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://vm-centos-cluster-ers.sosiouxme.lan:8210,vm-centos-cluster-ers.sosiouxme.lan,8210, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={100 101 108 116 97 45 99 108 117 …(13)}, ]. This operation will timeout if no session state has been received within 60 seconds.
13-Aug-2010 22:26:16.635 SEVERE org.apache.catalina.ha.session.DeltaManager.waitForSendAllSessions Manager [localhost#/manager]: No session state send at 8/13/10 10:25 PM received, timing out after 60,009 ms.

And so on for the rest of my webapps too! And if I try to access my cluster, it seems to be hung! While I was writing this up I think I figured out the problem with that. On the first node, I had gone into the context files for the manager and host-manager apps and set distributable=”false” (doesn’t really make sense to distribute manager apps). On the second node I had not done the same. My bad, BUT:

1. Why did it take 60 seconds to figure this out for EACH app; and
2. Why did EVERY app, not just the non-distributable ones, fail replication (at 60 seconds apiece)?

Well, having cleared that up my cluster with DeltaManager session replication seems to be working great.

Cluster with BackupManager, multicast

OK, here’s the surprise denouement – this seems to have just worked out of the box. I didn’t even think multicast would work without me having to tweak something in my OS (CentOS 5.5) or my router. But it did, and failover worked flawlessly when I killed a node too. Sweet.