OpenShift v2 cartridges: node host tools

There is a series starting on the OpenShift blog about the v2 cartridge format. Check it out. Way more official than whatever I write here.

Updated 2013-04-17 – updates marked below.

I introduced v2 cartridges in a previous post. If you have an OpenShift Origin node host running and you’ve toggled it to v2 mode, follow along.

When a user creates an application, that flows through the broker and mcollective to the node via the MCollective openshift.rb agent. You can shortcut that path if you want to create gears and configure cartridges into them more directly on the node host. None of the following involves the broker (so, of course, the broker will deny all knowledge of it if you ask).

Creating a gear

You can use the oo-app-create command on a node host to create a gear arbitrarily. Several parameters are required. You can run it with -h to see all the options. The main things you need are:

  1. An application UUID. This is a unique string that identifies the whole application; if the app has multiple gears, all will still share the application ID. Once the gear is created this goes in an environment variable.
  2. A gear UUID, which is referred to as a “container” ID. This is a unique string that identifies the gear. For single-gear apps, we typically just re-use the application ID, but that’s just convention.
  3. The application name – this would ordinarily be what a developer called their application.
  4. A namespace, which would ordinarily be the developer’s “domain”.

So the fun news is, since you don’t have to deal with the broker, you can totally make these up. There are a few requirements:

  1. The UUIDs do actually have to be unique, at least on the node host (they’re supposed to be unique across OpenShift). The broker just makes these up and instructs the node to use them.
  2. Gear UUID and name and namespace need to be all “word” characters. App name and UUID can be basically anything.
  3. Gear UUID will be used to create a system user, so it can’t violate the restrictions on that – e.g. it can’t be too long.

So, once you’ve made up your values, you can just create an empty gear like so:

# UUID=9vr98yvfr98ju23vfjuf
# DOMAIN=lm20130412
# NAME=testname
# oo-app-create --with-app-uuid $UUID \
--with-container-uuid $UUID \
--with-namespace $DOMAIN \
--with-app-name $NAME

Once you’ve done this, you check that the gear has been created with “id $UUID” and by looking in /var/lib/openshift for a directory of the same name.

A quick description of some of the optional parameters is in order:

  • --with-container-name is the name for the gear as opposed to the app – it just defaults to the app name if not specified. This is what is combined with the domain to create the DNS record for your gear – even if it’s a child gear in a scaled app, it will get its own DNS entry (although if you’re manually creating gears this way, the broker never knows to create the DNS entry so it’s rather moot).
  • --with-uid is used to specify a numeric user ID for the gear user – this is specified by the broker for nodes that are in a district; the UID is chosen from a pool that is available in the district and reserved for the gear regardless of which node in the district it lands on. So, it’s specified at the time the gear is created. If not specified, the node just picks an available one.

Distinguishing v1 and v2 gears

Even before we’ve done anything else with the new gear, it is marked as a v2 gear. Look at the files in the gear:

├── app-root
│   ├── data
│   │   └── .bash_profile
│   ├── repo -> runtime/repo
│   └── runtime
│   ├── data -> ../data
│   ├── repo
│   └── .state
├── .env
│   ├── HISTFILE
│   ├── HOME
│   ├── PATH
│   ├── TMP
│   ├── TMPDIR
│   └── TMP_DIR
├── .sandbox
│   └── 9vr98yvfr98ju23vfjuf
├── .ssh
└── .tmp

There is exactly one difference from a v1 gear: the CARTRIDGE_VERSION_2 env var. But that’s enough – the presence of this addition is used to decide whether to use v1 or v2 logic with cartridges in this gear.

Configuring a cartridge into the gear

So, let’s actually add a cartridge. You can do this with the oo-cartridge command. This is basically a convenience wrapper for manual testing – nothing in the product uses this script, but it is an entry point to the same code that actually is executed via MCollective to instantiate a cartridge in the gear.

# oo-cartridge -a add -c $UUID -n mock-0.1 -v -d
Cartridge add succeeded
Creating version marker for 0.1

Although I’ve added the -v -d flags (verbose, debug) you can see there isn’t much output here. Without either flag you just get the first line (success or failure). The “verbose” flag adds the output from the start hook after the cartridge is added. The “debug” flag will give detailed output if there is an exception (otherwise it adds nothing). To see what is really going on, you’ll want to look at the platform logs.

The platform logs are configured in /etc/openshift/node.conf and located in /var/log/openshift/node/. I suggest for the purposes of understanding, set the platform.log level to INFO in order to understand the flow of what’s happening, and leave platform-trace.log at DEBUG level to consult for the actual bash commands and their results. If you were developing a cartridge, though, you’d probably want platform.log at DEBUG level (the default) to see the bash commands mixed in with the code-level flow.

Example INFO-level platform.log for the above (leaving off timestamps and such):

Creating cartridge directory 9vr98yvfr98ju23vfjuf/mock
Created cartridge directory 9vr98yvfr98ju23vfjuf/mock
Creating private endpoints for 9vr98yvfr98ju23vfjuf/mock
Created private endpoint for cart mock in gear 9vr98yvfr98ju23vfjuf: [OPENSHIFT_MOCK_EXAMPLE_IP1=, OPENSHIFT_MOCK_EXAMPLE_PORT1=8080]
Created private endpoint for cart mock in gear 9vr98yvfr98ju23vfjuf: [OPENSHIFT_MOCK_EXAMPLE_IP1=, OPENSHIFT_MOCK_EXAMPLE_PORT2=8081]
Created private endpoint for cart mock in gear 9vr98yvfr98ju23vfjuf: [OPENSHIFT_MOCK_EXAMPLE_IP1=, OPENSHIFT_MOCK_EXAMPLE_PORT3=8082]
Created private endpoint for cart mock in gear 9vr98yvfr98ju23vfjuf: [OPENSHIFT_MOCK_EXAMPLE_IP2=, OPENSHIFT_MOCK_EXAMPLE_PORT4=9090]
Created private endpoints for 9vr98yvfr98ju23vfjuf/mock
mock attempted lock/unlock on out-of-bounds entry [~/invalid_mock_locked_file]
Running setup for 9vr98yvfr98ju23vfjuf/mock
Ran setup for 9vr98yvfr98ju23vfjuf/mock
Creating gear repo for 9vr98yvfr98ju23vfjuf/mock from ``
Created gear repo for 9vr98yvfr98ju23vfjuf/mock
Processing ERB templates for /var/lib/openshift/9vr98yvfr98ju23vfjuf/mock/**
Connecting frontend mapping for 9vr98yvfr98ju23vfjuf/mock: => with options: {"websocket"=>true}
Connecting frontend mapping for 9vr98yvfr98ju23vfjuf/mock: /front1a => with options: {}
configure output: Creating version marker for 0.1

platform-trace.log DEBUG-level output for the tail end of that is:

oo_spawn running service openshift-node-web-proxy reload: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 12>, :err=>#<IO:fd 9>}
oo_spawn buffer(11/) Reloading node-web-proxy:
oo_spawn buffer(11/) [
oo_spawn buffer(11/) OK
oo_spawn buffer(11/) ]
oo_spawn buffer(11/)
oo_spawn buffer(11/)
oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130413-30162-z6g4cz/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 8>}
oo_spawn running /usr/sbin/httxt2dbm -f DB -i /etc/httpd/conf.d/openshift/nodes.txt -o /etc/httpd/conf.d/openshift/nodes.db-20130413-30162-82d3xc/new.db: {:unsetenv_others=>false, :close_others=>true, :in=>"/dev/null", :out=>#<IO:fd 11>, :err=>#<IO:fd 8>}

In OpenShift you can compose an application by adding multiple cartridges, e.g. database or cron cartridges. The mock-plugin cartridge tests this functionality. You can use oo-cartridge to add this as well:

# oo-cartridge -a add -c $UUID -n mock-plugin-0.1 -d
Cartridge add succeeded
Creating version marker for 0.1

You can check that your gear is up and running with curl. Your gear has been configured into the front-end proxy’s routing, even though its DNS record doesn’t exist. You can tell the proxy which gear to access by setting the host header:

curl http://localhost/ -IH "Host: $NAME-$DOMAIN.$CLOUD_DOMAIN"

(You can get CLOUD_DOMAIN from /etc/openshift/node.conf) Of course with the mock cartridge, there may not be much to see; another cartridge like php-5.3 or ruby-1.9 will have content by default.

Cucumber tests

The mock cartridge is a testing tool for putting the cartridge logic through its paces. Take a look at the cucumber tests in the origin-server repo to see that in action (the mock cartridge feature is controller/test/cucumber/cartridge-mock.feature).

Updated 04/17: even as I was writing this, the cartridge-mock.feature was split into platform-* features, e.g. platform-basic.feature. Look at those instead.

You can run these by checking out the origin-server git repo on your node host and running cucumber against the feature file you are interested in (of course you must have the cucumber gem installed):

# cucumber cartridge-mock.feature
Using RSpec 2
simple: 17816 ruby 1.9.3 2012-11-10 [x86_64-linux]
Feature: V2 SDK Mock Cartridge
Scenario: Exercise basic platform functionality in isolation # cartridge-mock.feature:4

If you were developing a new v2 cartridge, BDD with a cucumber feature would probably be a better approach than the manual testing this post is demonstrating, especially for testing composing and scaling.

Logging into the gear

Update 04/17: Adding this section

Now that you have a gear with a cartridge or two in it, you might want to log in and look around, like you are used to with normal gears. Of course this really just means getting a login as the gear user. Normally you would do that with ssh, but you haven’t set up an ssh key for the gear yet. It’s easy to do that, but why bother? You can just use su, right?

# su - $UUID
Invalid context: unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023, expected unconfined_u:system_r:openshift_t:s0:c0,c502

Not so fast. The gear runs in a specialized SELinux context, and normal su doesn’t handle that. For this purpose you need oo-su:

# oo-su $UUID -c oo-trap-user

This will get you an ordinary gear-user login (preceded by a few error messages that don’t seem to harm anything). oo-trap-user is the login shell; of course, you don’t have to do that, you can use oo-su similarly to directly run any command in the context of the gear user.


You can remove a cartridge from a gear in much the same way it was added:

# oo-cartridge -a delete -c $UUID -n mock-plugin-0.1 -d
Cartridge delete succeeded
# oo-cartridge -a delete -c $UUID -n mock-0.1 -d
Cartridge delete succeeded

You’ll notice, though, that the gear is likely not left pristine. Cartridges leave log files and more even after removed. The base framework cartridges are particularly bad about this. You’ll even find that removing one framework cartridge and adding another may cause a failure. That’s because in real usage, framework cartridges are never removed. The whole gear is simply discarded:

# oo-app-destroy -a $UUID -c $UUID --with-app-name $NAME --with-namespace $DOMAIN

So, that is the simplest cleanup.

Reverse proxy resource usage – httpd 2.2.15

Recently I had reason to find out what happens when your httpd config includes a LOT of ProxyPass statements. The particular use case involved an application farm with one VirtualHost per domain for hundreds of domains, each very similar (though not identical) with dozens of ProxyPass statements to shared backend apps. But just a few dozen domains resulted in very high memory and CPU use.

I set up a simple test rig (CentOS 5 VM,1GB RAM, single x86 CPU, ERS 4.0.2 httpd 2.2.15) and ran some unscientific tests where the config included 20,000 ProxyPass statements with these variables:

  1. Unique vs. repeated – unique statements each proxied to a unique destination, while repeated ones proxied different locations to the same destination.
  2. Balancer – the statements either proxy directly to the URL, or use a pre-defined balancer:// address.
  3. Vhost – the ProxyPass statements were either all in the main_server definition or inside vhosts, one per.
  4. MPM – either prefork or worker MPM is used.

No actual load was applied to the server – I just wanted to see what it took to read the config and start up. Settings were defaults per MPM (5 children for prefork, 3 children for  worker) – obviously you’d tune these depending on usage. I tried to wait for things to settle down a bit after startup before reading “top” sorted by memory usage.

I also tried some other methods for accomplishing this task to see what the memory footprint would be.

Continue reading

unit testing android

Trying out ScribeFire for blogging, looks decent (and is a plugin for FireFox so cross-platform). So far looks nice.

Looking at Diego Torres Milano’s blog: Android: Testing on the Android platform – Unit tests – obviously this is over a year old and out of date (1.0 was current then). When creating a project now the ADT plugin allows you to create a parallel test project at the same time or a later time. I somehow picked out the right way to run a simple unit test once, but it doesn’t seem to want to run again. And when running as an “Android unit test” it actually opens an emulator in order to do unit tests?!? I can kind of understand that but it seems unnecessary for unit testing. For now, writing this off as too out of date.

Continuing with Diego Torres Milano’s blog: Android: Testing on the Android platform – ApiDemos tests. I followed the directions – I think – but when I pasted the source files into the test project they got pasted in the original package locations (e.g. not, which seemed to cause no problems. Linking the test project to the original was as advertised. Directions are ambiguous as to which manifest file to copy to the test project – get the one from the test directory. I was able to run the ApiDemosTests package as an Android Junit Test and see it actually go; did not have to do anything in the emulator, and the results show up in the JUnit widget in eclipse. I could see things happening in the emulator, so I guess it makes sense to have it up :-) I also tried running them from the Dev Tools or from adb and that worked as specified. Good to know.

Diego didn’t label this post so I almost missed it: Diego Torres Milano’s blog: Android: Testing on the Android platform – Creating tests; and this is a pretty good starting point. Got my test suite runner and a unit test shell set up, going to have to stop here for now.

So it’s great to see that this guy is also a bit frustrated with the docs (mostly, lack thereof) on testing; like emulator skins, tests are provided for you to reverse engineer, but not explained anywhere. I guess working tests are a little better than actual documentation which can get out of date fast. Diego seems to have a good understanding of JUnit and testing before coming to Android, which helps him. I don’t have that, but my brother does; this seems like an area that’s ripe for exploration and making a name by explaining. Heck, it would probably be worthwhile just to provide links to selected relevant posts by Diego putting them in an order that would be helpful to a beginner learning this stuff.

At my last meetup I told people that after you create an AVD, the tool doesn’t provide you any way to change its configuration. This is true, but you can edit the files that specify the config pretty easily – they aren’t even XML. The question is whether the emulator image itself (in the user’s .android directory) is modified at all according to config. I suspect the answer is “no.” At least, I tried changing the skin on an AVD (edit config.ini within the .avd definition) with no adverse consequences. But that’s superficial; other elements that refer to actual capabilities might need a matching setting inside the emulator image. I don’t know yet.

working together, minor discoveries

New Android from scratch blog post. Woo! Also, sent my merged G1 skin to the original author to see if he’ll put it all in the Creative Commons so we can derive from it :-) Also, I discovered exports iCal format, which of course means I can see my group’s calendar in my Google calendar! I love it when things work together.

I’d like to learn about testing and TDD for Android apps. Looks like Diego Torres knows a thing or two about this – surprisingly the SDK docs don’t touch on it AFAICS.

My new “Hello, Android” book introduces some stuff I had no idea about; e.g. I didn’t know you could use resources to define menus and preferences interfaces, and I didn’t know that you could specify different layout files for portrait/landscape and not have to futz with that kind of magic in code. I wonder if you can go further and specify different layout files for different resolutions? That would make sense. Anyway, resources are not just to keep strings and layout out of code, and not just for I18N, looks like they help with lots of platform stuff.