The OpenShift cartridge refactor: a brief introduction

If you’re watching the commit logs over at OpenShift Origin you’ll see a lot of activity around “v2” cartridges (especially a lot of “WIP” commits). For a variety of reasons we’re refactoring cartridges to make it easier to write and maintain them. We’re particularly interested in enabling those who wish to write cartridges, and part of that includes removing as much as possible from the current cartridge code that is really generic platform code and shouldn’t be boilerplate repeated in cartridges. And in general, we’re just trying to bring more sanity and remove opacity.

If you’ve fired up Origin lately you wouldn’t necessarily notice that anything has changed. The refactored cartridges are available in parallel with existing cartridges, and you have to opt in to use them. To do that, use the following command as root on a node host:

# oo-cart-version -c toggle
Node is currently in v1 mode
Switching node cartridge version
Node is currently in v2 mode

The node now works with the cartridges installed in /usr/libexec/openshift/cartridges/v2 (rather than the “v1” cartridges in /usr/libexec/openshift/cartridges – BTW these locations are likely to change, watch the RPM packaging for clues). Aside from the separate cartridge location, there are logic branches for the two formats in the node model objects, most prominently in OpenShift::ApplicationContainer (application_container.rb under the openshift-origin-node gem) making a lot of calls against @cartridge_model which is either a V1CartridgeModel or a V2CartridgeModel object depending.

The logic branches are based on two things – for an existing gear, the cartridge format already present is used; otherwise, for new gears, the presence of a marker file /var/lib/openshift/.settings/v2_cartridge_format is checked (which is the main thing the command above changes) – if present, use v2 cartridges, otherwise use the old ones. In this way, the development and testing of v2 cartridges can continue without needing a fork / branch and without disrupting the use of v1 cartridges.

A word of warning, though: you can use gears with the v1 and v2 cartridges in parallel on the same node (toggle back and forth), but don’t try to configure an embedded cart from one format into a gear with the other. Also, do not set a different mode on different nodes in the same installation. Results of trying to mix and match that way are undefined, which is to say, probably super broken.

Let’s look around a bit.

# ls /usr/libexec/openshift/cartridges/
10gen-mms-agent-0.1 diy-0.1 jbossews-1.0 mongodb-2.2 phpmyadmin-3.4 rockmongo-1.1 zend-5.6
abstract embedded jbossews-2.0 mysql-5.1 postgresql-8.4 ruby-1.8
abstract-httpd haproxy-1.4 jenkins-1.4 nodejs-0.6 python-2.6 ruby-1.9
abstract-jboss jbossas-7 jenkins-client-1.4 perl-5.10 python-2.7 switchyard-0.6
cron-1.4 jbosseap-6.0 metrics-0.1 php-5.3 python-3.3 v2

# ls /usr/libexec/openshift/cartridges/v2
diy haproxy jbosseap jbossews jenkins jenkins-client mock mock-plugin mysql perl php python ruby

There look to be a lot fewer cartridges under v2, and that’s not just because they’re not all complete yet. Notice what’s missing in v2? Version numbers. You’ll see the same thing looking in the source at the cartridge source trees and package specs; you don’t have a single cartridge per version anymore. It’s possible to support multiple different runtimes from the same cartridge. This is evident if you look in the ruby cartridge. First, there’s the cartridge manifest:

# grep Version /usr/libexec/openshift/cartridges/v2/ruby/metadata/manifest.yml
Version: '1.9'
Versions: ['1.9', '1.8']
Cartridge-Version: 0.0.1

There’s a default version if none is specified when configuring the cartridge, but there are two versions available in the same cartridge. Also notice the separate directories for version-specific implementations:

# ls /usr/libexec/openshift/cartridges/v2/ruby/versions/
1.8 1.9 shared

So rather than have completely separate cartridges for the different versions, different versions can live in the same cartridge and directly share the things they have in common, while overriding the usually-minor differences. This doesn’t mean we’re going to see ruby versions 1.9.1, 1.9.2, 1.9.3, etc. – in general you’ll only want one current version of a supported branch, such that security and bug fixes can be applied without having to migrate apps to a new version. But it means we cut down on a lot of duplication of effort for multi-versioned platforms. We can put ruby 1.8, 1.9, and 2.0 all in one cartridge and share most of the cartridge code.

You might be wondering how to specify which version you get. I’m not sure what is planned for the future, but at this time I don’t believe the logic branches for v2 cartridges have been extended to the broker. Right now, if you look in /var/log/mcollective.log for the cartridge-list action, you’ll see the node is reporting two separate Ruby cartridges just like before, which are reported back to the client, and you still request app creation with the version in the cartridge:

$ rhc setup
...
Run 'rhc app create' to create your first application.
Do-It-Yourself rhc app create <app name> diy-0.1
 JBoss Enterprise Application Platform rhc app create <app name> jbosseap-6.0.1
 Jenkins Server rhc app create <app name> jenkins-1.4
 Mock Cartridge rhc app create <app name> mock-0.1
 PHP 5.3 rhc app create <app name> php-5.3
 Perl 5.10 rhc app create <app name> perl-5.10
 Python 2.6 rhc app create <app name> python-2.6
 Ruby rhc app create <app name> ruby-1.9
 Ruby rhc app create <app name> ruby-1.8
 Tomcat 7 (JBoss EWS 2.0) rhc app create <app name> jbossews-2.0
$ rhc app create rb ruby-1.8
...
Application rb was created.

If you look in v2_cart_model.rb, you’ll see there’s a FIXME that parses out the version from the base cart name to handle this – the FIXME is to note that this should really be specified explicitly in an updated node command protocol. So at this time, there’s no broker-side API change to pick which version from a cartridge you want. But look for that to change when v2 carts are close to prime time.

By the way, if you’re used to looking in /var/log/mcollective.log to see broker/node interaction, that’s still there (you probably want to set loglevel = info in /etc/mcollective/server.cfg) but a lot more details about the node actions that result from these requests are now recorded in /var/log/openshift/node/platform.log (location configured in /etc/openshift/node.conf). You can watch this to see exactly how mcollective actions translate into system commands, and use this to manually test actions against developing cartridges (see also the mock cartridge and the test cases against it).

You’ll notice if you follow some cartridge actions (e.g. “restart”) through the code that the v2 format has centralized a lot of functions into a few scripts. Where before, each action and hook resulted to a call to a separate script (often symlinked in from the “abstract” cartridge which anyone would admit, is kind of a hack):

# ls /usr/libexec/openshift/cartridges/ruby-1.8/info/{bin,hooks}

/usr/libexec/openshift/cartridges/ruby-1.8/info/bin:
app_ctl.sh build.sh post_deploy.sh ps threaddump.sh
app_ctl_stop.sh deploy_httpd_config.sh pre_build.sh sync_gears.sh
/usr/libexec/openshift/cartridges/ruby-1.8/info/hooks:
add-module deploy-httpd-proxy reload restart stop tidy
configure info remove-httpd-proxy start system-messages update-namespace
deconfigure move remove-module status threaddump

In the new format, these are just options on a few scripts:

# ls /usr/libexec/openshift/cartridges/v2/ruby/bin/
build control setup teardown

If you look at the mcollective requests and the code, you’ll see the requests haven’t changed, but the v2 code is just routing it to the new scripts. For instance, “restart” is now just an option to the “control” script above.

Those are just some of the changes that are in the works. The details are still evolving daily, too fast for me to keep track of frankly, but if you’re interested in what’s happening, especially interested in writing cartridges for OpenShift, you might like to dive into the existing documentation describing the new format:

https://github.com/openshift/origin-server/blob/master/node/README.writing_cartridges.md

Other documents in the same directory may or may not distinguish between v1 and v2 usage, but regardless should be useful, if sometimes out of date, reading.

Advertisements