Fixing Apache httpd reverse proxy redirect rewrites

ProxyPassReverse statement was adding an extra “/” in its Location: header rewrites. I noticed this when requesting a URL like “/petcare”. Tomcat would redirect this appropriately with a 302 and this header:

Location: http://vm-centos-cluster-ers.sosiouxme.lan:8100/petcare/

But when it came back through the proxy as “/http/nocluster/petcare”, it ended up rewritten as:

Location: http://vm-centos-cluster-ers.sosiouxme.lan/http/nocluster//petcare/

It seemed like a small thing – and, after all, it still worked due to URL canonicalization – but I wanted to understand why this happened and make it right. Here’s a typical configuration section initially:

# use mod_proxy_http to connect to non-replicated tc instances
<Proxy balancer://http-nocluster/>
BalancerMember http://vm-centos-cluster-ers.sosiouxme.lan:8100 route=ers
BalancerMember http://vm-centos-cluster-tcs.sosiouxme.lan:8100 route=tcs
ProxySet stickysession=JSESSIONID nofailover=On
</Proxy>

<Location /http/nocluster/>
ProxyPass balancer://http-nocluster/
ProxyPassReverse balancer://http-nocluster/
ProxyPassReverseCookiePath / /http/nocluster/
ProxyHTMLURLMap / /http/nocluster/
</Location>

I figured it was just a matter of juggling where “/” appeared at the end of various things. I cranked up the logging to “debug” and tried a few changes one by one.

• Remove “/” from end of ProxyPass. This gave me a lovely 500 error and log messages:

ProxyPass balancer://http-nocluster

[debug] mod_proxy_balancer.c(46): proxy: BALANCER: canonicalising URL //http-noclusterpetcare

[debug] proxy_util.c(1525): [client 172.31.1.52] proxy: *: found reverse proxy worker for balancer://http-noclusterpetcare/

[…]

[warn] proxy: No protocol handler was valid for the URL /http/nocluster/petcare. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

• Remove “/” from end of ProxyPassReverse. No apparent effect.
• Remove “/” from end of <Proxy balancer://http-nocluster/> – no effect.
• Remove “/” from end of <Location /http/nocluster/> – now we were getting somewhere! The Location header was rewritten correctly; only problem is that after rewriting, it was passed through to Tomcat as //petcare/ and failing.
• Remove “/” from the end of everything! This seems to be what works best – everything passes through correctly and Location is rewritten correctly. So the configuration I ended up with is:

# use mod_proxy_http to connect to non-replicated tc instances
<Proxy balancer://http-nocluster>
BalancerMember http://vm-centos-cluster-ers.sosiouxme.lan:8100 route=ers
BalancerMember http://vm-centos-cluster-tcs.sosiouxme.lan:8100 route=tcs
ProxySet stickysession=JSESSIONID nofailover=On
</Proxy>

<Location /http/nocluster>
ProxyPass balancer://http-nocluster
ProxyPassReverse balancer://http-nocluster
ProxyPassReverseCookiePath / /http/nocluster/
ProxyHTMLURLMap / /http/nocluster/
</Location>

This was pretty much the only thing I tried that worked properly. Now, with mod_proxy_ajp it was a different story. The configuration looked pretty similar (because I’d done a cut/paste/edit):

# use mod_proxy_ajp to connect to non-replicated tc instances
<Proxy balancer://ajp-nocluster>
BalancerMember ajp://vm-centos-cluster-tcs.sosiouxme.lan:8109 route=tcs
BalancerMember ajp://vm-centos-cluster-ers.sosiouxme.lan:8109 route=ers
ProxySet stickysession=JSESSIONID nofailover=On
</Proxy>

<Location /ajp/nocluster/>
ProxyPass balancer://ajp-nocluster/
ProxyPassReverse balancer://ajp-nocluster/
ProxyPassReverseCookiePath / /ajp/nocluster/
ProxyHTMLURLMap / /ajp/nocluster/
</Location>

Thing is, my ProxyPassReverse there wasn’t doing anything at all. This is a little-known fact about how mod_proxy_ajp and ProxyPassReverse interact: an AJP connection doesn’t get a new http request to the backend; rather the HTTP headers from the request to the proxy are passed to the backend, and typically presented by Tomcat to the app as the request headers. So when the app (or Tomcat) forms a redirect (Location: header), it is relative to the host and port on the proxy, not the backend.

Meanwhile, ProxyPassReverse is very literal-minded. It only matches exactly what you put in the statement. So it’s a common error to have config like this:

ProxyPass / ajp://backend.example.com:8009/

ProxyPassReverse / ajp://backend.example.com:8009/

The ProxyPassReverse there isn’t doing anything at all, because it’s never going to see a “Location: ajp://backend.example.com:8009/” header from the backend – instead it will see URLs based on the front end. Most people won’t notice this because most people are using the same paths on front and backend, so nothing needed to be rewritten anyway. I had to be different and remap paths so I noticed when they weren’t rewritten.

The exception to the literal-mindedness of PPR is the balancer:// faux protocol. When you have a bunch of http backends in a balancer, you would normally need to rewrite headers corresponding to any of them – so, a PPR directive for each. This is pretty tedious. Starting in (I think) httpd 2.2.9 you could do a single PPR directive with the balancer:// notation as above and get this for free. I was hoping it would be smarter about AJP, but it’s not. That’s not such a big deal, though – since the host and port are always that of the front-end, I only need a single PPR for the rewrite.

<Location /ajp/nocluster/>
ProxyPass balancer://ajp-nocluster/

# note: http! This is the proxy server URL
ProxyPassReverse http://vm-centos-cluster-ers.sosiouxme.lan/
ProxyPassReverseCookiePath / /ajp/nocluster/
ProxyHTMLURLMap / /ajp/nocluster/
</Location>

And I didn’t even have to futz with the slashes, it just worked with them in.