Fixing Apache httpd reverse proxy redirect rewrites

ProxyPassReverse statement was adding an extra “/” in its Location: header rewrites. I noticed this when requesting a URL like “/petcare”. Tomcat would redirect this appropriately with a 302 and this header:

Location: http://vm-centos-cluster-ers.sosiouxme.lan:8100/petcare/

But when it came back through the proxy as “/http/nocluster/petcare”, it ended up rewritten as:

Location: http://vm-centos-cluster-ers.sosiouxme.lan/http/nocluster//petcare/

It seemed like a small thing – and, after all, it still worked due to URL canonicalization – but I wanted to understand why this happened and make it right. Here’s a typical configuration section initially:

# use mod_proxy_http to connect to non-replicated tc instances
<Proxy balancer://http-nocluster/>
BalancerMember http://vm-centos-cluster-ers.sosiouxme.lan:8100 route=ers
BalancerMember http://vm-centos-cluster-tcs.sosiouxme.lan:8100 route=tcs
ProxySet stickysession=JSESSIONID nofailover=On

<Location /http/nocluster/>
ProxyPass balancer://http-nocluster/
ProxyPassReverse balancer://http-nocluster/
ProxyPassReverseCookiePath / /http/nocluster/
ProxyHTMLURLMap / /http/nocluster/

I figured it was just a matter of juggling where “/” appeared at the end of various things. I cranked up the logging to “debug” and tried a few changes one by one.

  • Remove “/” from end of ProxyPass. This gave me a lovely 500 error and log messages:

ProxyPass balancer://http-nocluster

[debug] mod_proxy_balancer.c(46): proxy: BALANCER: canonicalising URL //http-noclusterpetcare

[debug] proxy_util.c(1525): [client] proxy: *: found reverse proxy worker for balancer://http-noclusterpetcare/


[warn] proxy: No protocol handler was valid for the URL /http/nocluster/petcare. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

  • Remove “/” from end of ProxyPassReverse. No apparent effect.
  • Remove “/” from end of <Proxy balancer://http-nocluster/> – no effect.
  • Remove “/” from end of <Location /http/nocluster/> – now we were getting somewhere! The Location header was rewritten correctly; only problem is that after rewriting, it was passed through to Tomcat as //petcare/ and failing.
  • Remove “/” from the end of everything! This seems to be what works best – everything passes through correctly and Location is rewritten correctly. So the configuration I ended up with is:

# use mod_proxy_http to connect to non-replicated tc instances
<Proxy balancer://http-nocluster>
BalancerMember http://vm-centos-cluster-ers.sosiouxme.lan:8100 route=ers
BalancerMember http://vm-centos-cluster-tcs.sosiouxme.lan:8100 route=tcs
ProxySet stickysession=JSESSIONID nofailover=On

<Location /http/nocluster>
ProxyPass balancer://http-nocluster
ProxyPassReverse balancer://http-nocluster
ProxyPassReverseCookiePath / /http/nocluster/
ProxyHTMLURLMap / /http/nocluster/

This was pretty much the only thing I tried that worked properly. Now, with mod_proxy_ajp it was a different story. The configuration looked pretty similar (because I’d done a cut/paste/edit):

# use mod_proxy_ajp to connect to non-replicated tc instances
<Proxy balancer://ajp-nocluster>
BalancerMember ajp://vm-centos-cluster-tcs.sosiouxme.lan:8109 route=tcs
BalancerMember ajp://vm-centos-cluster-ers.sosiouxme.lan:8109 route=ers
ProxySet stickysession=JSESSIONID nofailover=On

<Location /ajp/nocluster/>
ProxyPass balancer://ajp-nocluster/
ProxyPassReverse balancer://ajp-nocluster/
ProxyPassReverseCookiePath / /ajp/nocluster/
ProxyHTMLURLMap / /ajp/nocluster/

Thing is, my ProxyPassReverse there wasn’t doing anything at all. This is a little-known fact about how mod_proxy_ajp and ProxyPassReverse interact: an AJP connection doesn’t get a new http request to the backend; rather the HTTP headers from the request to the proxy are passed to the backend, and typically presented by Tomcat to the app as the request headers. So when the app (or Tomcat) forms a redirect (Location: header), it is relative to the host and port on the proxy, not the backend.

Meanwhile, ProxyPassReverse is very literal-minded. It only matches exactly what you put in the statement. So it’s a common error to have config like this:

ProxyPass / ajp://

ProxyPassReverse / ajp://

The ProxyPassReverse there isn’t doing anything at all, because it’s never going to see a “Location: ajp://” header from the backend – instead it will see URLs based on the front end. Most people won’t notice this because most people are using the same paths on front and backend, so nothing needed to be rewritten anyway. I had to be different and remap paths so I noticed when they weren’t rewritten.

The exception to the literal-mindedness of PPR is the balancer:// faux protocol. When you have a bunch of http backends in a balancer, you would normally need to rewrite headers corresponding to any of them – so, a PPR directive for each. This is pretty tedious. Starting in (I think) httpd 2.2.9 you could do a single PPR directive with the balancer:// notation as above and get this for free. I was hoping it would be smarter about AJP, but it’s not. That’s not such a big deal, though – since the host and port are always that of the front-end, I only need a single PPR for the rewrite.

<Location /ajp/nocluster/>
ProxyPass balancer://ajp-nocluster/

# note: http! This is the proxy server URL
ProxyPassReverse http://vm-centos-cluster-ers.sosiouxme.lan/
ProxyPassReverseCookiePath / /ajp/nocluster/
ProxyHTMLURLMap / /ajp/nocluster/

And I didn’t even have to futz with the slashes, it just worked with them in.

Clustering Tomcat

I’ve been meaning for some time to set up Tomcat clustering in several different ways just to see how it works. I finally got a chance today. There are guides and other bits of information elsewhere on how to do all this. I’m not gonna tell you how, sorry; the point of this post is to document my problems and how I overcame them.

A word about setup

My front end load balancer is Apache httpd 2.2.15 (from SpringSource ERS 4.0.2) using mod_proxy_http, mod_proxy_ajp, mod_jk, and (just to keep things interesting) mod_rewrite as connectors to the backend Tomcat 6.0.28 instances. I wanted to try out 2-node Tomcat clusters (one side of each cluster ERS Tomcat, the other side tc Server 2.0.3) without any session replication (so, sticky sessions and you get a new session if there’s a failure) and with session replication via the standard Delta manager (which replicates all sessions to all nodes) and the Backup manager (which replicates all sessions to a single reference node, the “backup” for each app).  Basically the idea is to test out every conceivable way to put together a cluster with standard SpringSource technologies.

The first trick was mapping all of these into the  URL space for my httpd proxy. I wanted to put each setup behind its own URL /<connector>/<cluster> so e.g /http/backup and /ajp/delta. This is typically not done and for good reason; mod_proxy will let you do the mapping and will even clean up redirects and cookie paths from the backend, but to take care of self-referential links in the backend apps you actually have to rewrite the content that comes back; for that I installed mod_proxy_html, a non-standard module for doing such things. The reason it’s a non-standard module is that this approach is fraught with danger. But given that I mostly don’t care about how well the apps work in my demo cluster, I thought it’d be a great time to put it through its paces.

For this reason, most people make sure the URLs on the front-end and back-end are the same; and in fact, as far as I could tell, there was no way at all to make mod_jk do any mapping, so I’m setting it up as a special case – more later if it’s relevant. The best way to do this demo would probably be to set up virtual hosts on the proxy for each scenario and not require any URI mapping; if I run into enough problems I’ll probably fall back to that.

Problem 1: the AJP connector

I got things running without session replication fairly easily. My first big error with session replication was actually something else, but at first I thought it might be related to this warning in the Tomcat log:

Aug 13, 2010 9:43:00 PM org.apache.catalina.core.AprLifecycleListener init

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/jdk1.6.0_21/jre/lib/i386/server:/usr/java/jdk1.6.0_21/jre/lib/i386:/usr/java/jdk1.6.0_21/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
Aug 13, 2010 9:43:00 PM org.apache.catalina.startup.ConnectorCreateRule _setExecutor
WARNING: Connector [org.apache.catalina.connector.Connector@1d6f122] does not support external executors. Method setExecutor(java.util.concurrent.Executor) not found.

These actually turn out to be related:

<Listener className=”org.apache.catalina.core.AprLifecycleListener” SSLEngine=”on” />

<Connector  executor=”tomcatThreadPool”

I don’t really know why the APR isn’t working properly, but a little searching turned up some obscure facts: if the APR isn’t loaded, then for the AJP connector Tomcat makes a default choice of implementation that doesn’t use the executor thread pool. So you have to explicitly set the class to use like this:

<Connector  executor=”tomcatThreadPool”

Nice, eh? OK. But that was just an annoyance.

Problem 2: MBeans blowup

The real thing holding me back was this error:

Aug 13, 2010 9:52:16 PM org.apache.catalina.mbeans.ServerLifecycleListener createMBeans
SEVERE: createMBeans: Throwable
at org.apache.catalina.mbeans.MBeanUtils.createObjectName(
at org.apache.catalina.mbeans.MBeanUtils.createMBean(

Now what the heck was that all about? Well, I found no love on Google. But I did eventually guess what the problem was. This probably underscores a good methodology: when working on configs, add one thing at a time and test it out before going on. If only Tomcat had a “configtest” like httpd – it takes forever to “try it out”.

In case anyone else runs into this, I’ll tell you what it was. The Cluster Howto made it pretty clear that you need to set your webapps context to be distributable for clustering to work. It wasn’t clear to me where to put that, but I didn’t want to create a context descriptor for each app on each instance. I knew you could put a <Context> element in server.xml so that’s right where I put it, right inside the <Host> element:

<Context distributable=”true” />

Well, that turns out to be a bad idea. It causes the error above. So don’t do that. For the products I’m using, there’s a single context.xml that applies to all apps on the server; that’s where you want to put the distributable attribute.

Cluster membership – static

My next task was to get the cluster members in each case to recognize each other and replicate sessions. Although all the examples use multicast to do this, I wanted to try setting up static membership, because I didn’t want to look up how to enable multicast just yet. And it should be simpler, right? Well, I had a heck of a time finding this, but it looks like the StaticMembershipInterceptor is the path.

My interceptor looks like this:

<Interceptor className=””>
<Member className=”org.apache.catalina.tribes.membership.StaticMember”
port=”8210″ securePort=”-1″

(with the “host” being the other host in the 2-node cluster on each side). Starting with this configuration brings an annoying warning message:

WARNING: [SetPropertiesRule]{Server/Service/Engine/Cluster/Channel/Interceptor/Member} Setting property ‘uniqueId’ to ‘{1,2,3,4,5,6,7,8,9,0}’ did not find a matching property.

So I guess that property has been removed and the docs not updated; didn’t seem necessary anyway given the host/port combo should be unique.

In any case, the log at first looks encouraging:

Aug 13, 2010 10:13:04 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://vm-centos-cluster-tcs.sosiouxme.lan:8210,vm-centos-cluster-tcs.sosiouxme.lan,8210, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={100 101 108 116 97 45 99 108 117 …(13)}, ]
Aug 13, 2010 10:13:04 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:4
Aug 13, 2010 10:13:04 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{172, 31, 1, 108}:8210,{172, 31, 1, 108},8210, alive=25488670,id={66 108 -53 22 -38 -64 76 -110 -110 -54 28 -11 -126 -44 66 28 }, payload={}, command={}, domain={}, ]

Sounds like the other cluster member is found, right? I don’t know why it’s in there twice (once w/ hostname, once with IP) but I think the first is the configuration value, and the second is for when the member is actually contacted (alive=).

And indeed, for one node, later in the log as the applications are being deployed, I see the session replication appears to happen for each:

WARNING: Manager [localhost#/petclinic], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://vm-centos-cluster-tcs.sosiouxme.lan:8210,vm-centos-cluster-tcs.sosiouxme.lan,8210, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={100 101 108 116 97 45 99 108 117 …(13)}, ]. This operation will timeout if no session state has been received within 60 seconds.
Aug 13, 2010 10:13:09 PM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions
INFO: Manager [localhost#/petclinic]; session state send at 8/13/10 10:13 PM received in 186 ms.

Um, why is that a WARNING? Looks like normal operation to me, surely it should be an INFO. Whatever. The real problem is on the other side of the node:

13-Aug-2010 22:25:16.624 INFO org.apache.catalina.ha.session.DeltaManager.start Register manager /manager to cluster element Engine with name Catalina
13-Aug-2010 22:25:16.624 INFO org.apache.catalina.ha.session.DeltaManager.start Starting clustering manager at/manager
13-Aug-2010 22:25:16.627 WARNING org.apache.catalina.ha.session.DeltaManager.getAllClusterSessions Manager [localhost#/manager], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://vm-centos-cluster-ers.sosiouxme.lan:8210,vm-centos-cluster-ers.sosiouxme.lan,8210, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={100 101 108 116 97 45 99 108 117 …(13)}, ]. This operation will timeout if no session state has been received within 60 seconds.
13-Aug-2010 22:26:16.635 SEVERE org.apache.catalina.ha.session.DeltaManager.waitForSendAllSessions Manager [localhost#/manager]: No session state send at 8/13/10 10:25 PM received, timing out after 60,009 ms.

And so on for the rest of my webapps too! And if I try to access my cluster, it seems to be hung! While I was writing this up I think I figured out the problem with that. On the first node, I had gone into the context files for the manager and host-manager apps and set distributable=”false” (doesn’t really make sense to distribute manager apps). On the second node I had not done the same. My bad, BUT:

  1. Why did it take 60 seconds to figure this out for EACH app; and
  2. Why did EVERY app, not just the non-distributable ones, fail replication (at 60 seconds apiece)?

Well, having cleared that up my cluster with DeltaManager session replication seems to be working great.

Cluster with BackupManager, multicast

OK, here’s the surprise denouement – this seems to have just worked out of the box. I didn’t even think multicast would work without me having to tweak something in my OS (CentOS 5.5) or my router. But it did, and failover worked flawlessly when I killed a node too. Sweet.