Customizing OpenShift JBoss confs without customizing the cartridge

I added a feature recently to enable OpenShift administrators to specify (at the broker) a custom location to get the default app git template from. This allows you to customize the initial experience of developers when they create an app; so you can, for example, put your organization’s name and logo on them. This should be out in Origin nightly builds now and the Enterprise 2.0.3 point release coming soon.

For JBoss applications, there is an added use for this feature. JBoss configuration files are located in the application git repository, so if you wanted to change the default confs for these cartridges as an administrator, say to add a custom valve, you can. Users are free to ignore this, of course, either by specifying a different source for their code or blowing your changes away after creating the app. Still, it can be useful to set the defaults the way you like, and with this feature, you don’t have to customize the cartridge to do it. You just need to maintain a custom git repository.

There’s a slight complication, though, as I discovered when trying to demonstrate this. The JBoss cartridges construct configuration files with three processing steps in between the source and the outcome. These are:

  1. The “install” step of cartridge instantiation modifies the Maven pom.xml that ships with the default template, replacing strategically-placed {APP_NAME} entries with the application name. If you construct your template using the source, Maven will not like it if you leave these as-is.
  2. The “setup” step of cartridge instantiation combines shared configuration files with version-specific configuration files from the cartridge source.
  3. Most of the conf files in the application git repo are directly symlinked from the actual gear configuration. However, there are a few that aren’t, which happen to be the ones you tend to want to change. These are actually templates that are processed during every build of the application (i.e. every git push).

These aren’t hard to work around, but they’re a little surprising if you don’t know about them. Let me demonstrate how I would do this with an example. Let’s say we wanted to change the log format on a JBoss EWS 2.0 cartridge.

  1. First, create an EWS 2.0 app with the installed default:
    • rhc app create template jbossews-2.0
  2. Now edit the resulting “template” directory that git creates as needed:
    • Change .openshift/config/server.xml log valve as follows:
      <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
             prefix="localhost_access_log." suffix=".txt"
             pattern="CHANGED %h %l %u %t &quot;%r&quot; %s %b" />
    • Note, this is one of the files that is interpreted with every git push. The Connector element has expressions in it which are evaluated at build time on the gear.
    • Edit the pom.xml file. This is optional, but you may want to use a different groupId, artifactId, etc. than just the “template” app name. It’s possible to use env vars here, e.g.
      <groupId>${env.OPENSHIFT_APP_DNS}</groupId>

      … however, Maven will give out WARNINGs with every build and threatens to break in the future if you do this, so I don’t recommend it.

    • Commit the changes.
       git commit -am "Template customizations"
  3. Now put this git repo somewhere all the nodes can see it. You can put it on github if you like, or your internal gitolite instance, or just on a plain web server. For simplicity, I just put it directly on the node filesystem, but remember that all nodes have to have the template available in the same place (although it could be useful to vary the template contents according to gear profile):
    # mkdir -p /etc/openshift/templates
    # git clone template /etc/openshift/templates/jbossews2.git
  4. Now modify the broker to specify this as the default. In /etc/openshift/broker.conf:
    DEFAULT_APP_TEMPLATES=jbossews-2.0|file:///etc/openshift/templates/jbossews2.git

    … and restart the broker:

    $ service openshift-broker restart

    Of course, with multiple brokers, you need to do this for all of them.

At this point, whenever you create a new jbossews-2.0 application, it will default to using your template and the changed access log format.

Stuff that should just work…

Had one of those Maven/Eclipse experiences that was so infuriating, I need to record it here to make sure I can fix it easily next time.

Using STS 2.9.1 I created a “Dynamic Web Module” project. For the runtime I targeted tc Server / Tomcat 6. Then I proceeded to do some rudimentary JSP stuff, only to find that the JSTL taglibs were not resolving. It seems the expectation is that these would be provided by the container. Under JBoss they probably would be, but not under Tomcat. (I guess it makes sense… if you have multiple apps in the container, just let each one bundle the version desired – fewer container dependencies).

Fine; so I added the maven nature and the jstl.jar dependency. At some point I did Maven > Update project configuration. Suddenly the class that I had defined to back a form is not found. Also I’m getting this really annoying project error:

Dynamic Web Module 3.0 requires Java 1.6 or newer. [...] Maven WTP Configuration Problem
One or more constraints have not been satisfied.

WTF? Of course STS/Eclipse are configured to use Java 1.6… but my project apparently isn’t. So I go change that, but it doesn’t fix that error, and any time I update project config with Maven, it’s back to using JRE 1.5 and my Java source files are no longer on the build path as source files.

Turns out (took longer to find than to tell about it) the Maven compiler plugin doesn’t use Eclipse settings and just imposes its own defaults, i.e. Java 5, unless otherwise configured by a POM. And since a “Dynamic Web Project” uses Servlet 3.0 it requires Java 6. Boom.

Easy to fix, though annoying that I have to and there isn’t some Eclipse setting for this. Just add under the top-level POM:

<build>
    <plugins>
       <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>2.1</version>
          <configuration>
             <source>1.6</source>
             <target>1.6</target>
          </configuration>
       </plugin>
    </plugins>
 </build>

(Cripes, WordPress, can I get a “code” formatting/paste option already?? “Preformatted” most certainly isn’t.)

Then have Maven update project config again and 1.6 is in play. Oh, and instead of using the “src” directory for my source, I went ahead and changed to src/main/java as Maven expects, so that future “config update” will pick that up.

What does a tech support geek do all day anyway?

I hate to let a calendar month go by with no post.

Though hey, I’m so far behind on reading my web comics, I don’t even know how far behind I am anymore. That’s when you know it’s serious.

Being a parent and keeping things from falling apart has pretty much taken up all my energy outside work at this point. It takes up a lot of time, too, but the real constraint is energy and enthusiasm. It’s hard to really get into anything significant that you know will probably be interrupted soon. Thus I haven’t even gotten around to checking out the Android-friendly ORMlite updates. Someday!

At least some of the arcane stuff I’ve been working on for my day job is publicly visible! Too bad VMware doesn’t really have a way for KB authors to sign our work, but here are a few bits of interest in the web server / application server world…

I have some in the pipeline to attempt to finally explain Tomcat auto-deployment and logging in a way that a non-expert can follow and use (I dare say the engineers are too close to their work). I should work on the official docs, but that’s not officially part of my goals… if I could get someone to pay me for it, I probably would. I think I have something of a gift for technical docs. It just takes me such a long time, I doubt anyone would consider it worth funding. At the same time, I don’t want to quit doing actual technical work in order to document it.

Maybe soon I’ll document my journey of trying to use Fedora 16 for my workstation (VPNs… virtualization… and dual monitors, oh my!)

Hash collision DoS

Have been dealing with this vulnerability a little bit. Amusingly, my old favorite Perl has had the fix for this for years – salt the hash randomly so an attacker can’t predict how your entries will hash. That’s really the only fix, because while you might be able to mitigate the specific case of hashing CGI parameters, anything that takes user input in any form from potentially malicious clients could be vulnerable. That’s a pretty wide use case.

Of course, if the bad guys don’t know how the processing of input is implemented, it will be tricky for them to find the hole to exploit. So I suppose blocking the specific method (as Tomcat did by limiting the number of parameters it will hash) serves to block opportunistic attacks. But it may still leave possibilities for those who are really determined to cause havoc with a specific site.

 

Clustering Tomcat (part III): the HTTPS connector

Refer to my earlier posts on the subject for background. Here are further explorations, not having much to do with clustering as it turns out, but everything to do with proxying.

In a number of situations you might want to set up encrypted communication between the proxy and backend. For this, Tomcat supplies an HTTPS connector (as far as I know, the only way to encrypt requests to Tomcat).

Connector setup

Setup is actually fairly simple with just a few surprises. Mainly, the protocol setting on the connector remains “HTTP/1.1” not some version of “HTTPS” – the protocol being the language spoken, and SSL encryption being a layer on top of that which you specify with SSLProtocol. Basically, HTTPS connectors look like HTTP connectors with some extra SSL properties specifying the encryption:

<Connector executor="tomcatThreadPool"
 port="${https.port}"
 protocol="HTTP/1.1"
 connectionTimeout="20000"
 acceptCount="100"
 maxKeepAliveRequests="15"
 SSLEnabled="true"
 SSLProtocol="TLS"
 scheme="https"
 secure="true"
 keystorePass="changeit"
 clientAuth="false"
/>

If I really wanted to be thorough I would set clientAuth=”true” and set up the proxy with a client certificate signed by this server’s truststore, thereby guaranteeing only the proxy can even make a request. But not right now.

Note the “scheme” and “secure” properties here. These don’t actually affect the connection; instead, they specify what Tomcat should answer when a servlet asks questions about its request. Specifically, request.getScheme() is likely to be used to create self-referential URLs, while request.isSecure() is used to make several security-related decisions. Defaults are for a non-secure HTTP connector but they can be set to whatever makes sense in context – in fact, AFAICS the scheme can be set to anything and the connector will still serve data fine. Read on for clarity on uses of these properties. Continue reading

Clustering Tomcat (part II)

Refer to my earlier post on the subject for background. Here are some more things I ran into.

Tricking mod_proxy_html

It didn’t take too long to come up against something that mod_proxy_html wouldn’t rewrite correctly. In the petcare sample app, there’s a line in a Javascript file (resources/jquery/openid-selector/js/openid-jquery.js) that looks like this:

	img_path: '/petcare/resources/jquery/openid-selector/images/',

There’s nothing to identify it as a URL; it’s just a regular old JS property,  so mod_proxy_html does nothing with it. In fact, as far as I can see, there isn’t any way to tweak the module configuration to adjust it, even knowing exactly what the problem is. But having the path wrong meant in this case that some vital icons on the OpenID sign-in page didn’t show up.

So I went ahead and broke everything out into vhosts like I should have all along. Look folks, if you’re reverse proxying, you really just need to have the same path on the front and backend unless what you’re doing is dead simple. If your front-end URL-space dictates a particular path, move your app to that path on the backend; don’t try to remap the path, it will almost certainly cause you headaches.

<Location> bleeds into VirtualHost

Having created vhosts, I noticed something interesting: <Location> sections that I defined in the  main server config sometimes applied in the vhosts as well – sometimes not. In particular, applying a handler (like status-handler) also applied to vhosts, while a JkMount in the main section did not cause vhosts to also serve requests through mod_jk. I think there were some other oddities like this but I can’t recall them any more.

It’s worth noting that RewriteRule directives in the main server config are explicitly not inherited by vhosts unless you specify that they should be.

mod_jk cluster partitioning

mod_jk workers have this property “domain” that you can set (ref. the Tomcat Connector guide). It’s not exactly obvious what it does – it doesn’t make sense to use the domain as the route when you’ve already got a route for each instance. I also read somewhere that mod_jk can be used to partition a cluster. Reading between the lines a little bit and trying it out, here’s what I found:

If you specify a domain, the workers with the same domain will failover like a sub-cluster within the cluster; they all still have their own routes, but if one instance fails, mod_jk will try to route to another member of the same sub-cluster. This means that you only need to replicate sessions between the members of the sub-cluster (assuming you trust the sub-cluster not to fail entirely). This could significantly cut down on the amount of session replication network traffic in a large cluster.