Customizing OpenShift JBoss confs without customizing the cartridge

I added a feature recently to enable OpenShift administrators to specify (at the broker) a custom location to get the default app git template from. This allows you to customize the initial experience of developers when they create an app; so you can, for example, put your organization’s name and logo on them. This should be out in Origin nightly builds now and the Enterprise 2.0.3 point release coming soon.

For JBoss applications, there is an added use for this feature. JBoss configuration files are located in the application git repository, so if you wanted to change the default confs for these cartridges as an administrator, say to add a custom valve, you can. Users are free to ignore this, of course, either by specifying a different source for their code or blowing your changes away after creating the app. Still, it can be useful to set the defaults the way you like, and with this feature, you don’t have to customize the cartridge to do it. You just need to maintain a custom git repository.

There’s a slight complication, though, as I discovered when trying to demonstrate this. The JBoss cartridges construct configuration files with three processing steps in between the source and the outcome. These are:

  1. The “install” step of cartridge instantiation modifies the Maven pom.xml that ships with the default template, replacing strategically-placed {APP_NAME} entries with the application name. If you construct your template using the source, Maven will not like it if you leave these as-is.
  2. The “setup” step of cartridge instantiation combines shared configuration files with version-specific configuration files from the cartridge source.
  3. Most of the conf files in the application git repo are directly symlinked from the actual gear configuration. However, there are a few that aren’t, which happen to be the ones you tend to want to change. These are actually templates that are processed during every build of the application (i.e. every git push).

These aren’t hard to work around, but they’re a little surprising if you don’t know about them. Let me demonstrate how I would do this with an example. Let’s say we wanted to change the log format on a JBoss EWS 2.0 cartridge.

  1. First, create an EWS 2.0 app with the installed default:
    • rhc app create template jbossews-2.0
  2. Now edit the resulting “template” directory that git creates as needed:
    • Change .openshift/config/server.xml log valve as follows:
      <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
             prefix="localhost_access_log." suffix=".txt"
             pattern="CHANGED %h %l %u %t &quot;%r&quot; %s %b" />
    • Note, this is one of the files that is interpreted with every git push. The Connector element has expressions in it which are evaluated at build time on the gear.
    • Edit the pom.xml file. This is optional, but you may want to use a different groupId, artifactId, etc. than just the “template” app name. It’s possible to use env vars here, e.g.
      <groupId>${env.OPENSHIFT_APP_DNS}</groupId>

      … however, Maven will give out WARNINGs with every build and threatens to break in the future if you do this, so I don’t recommend it.

    • Commit the changes.
       git commit -am "Template customizations"
  3. Now put this git repo somewhere all the nodes can see it. You can put it on github if you like, or your internal gitolite instance, or just on a plain web server. For simplicity, I just put it directly on the node filesystem, but remember that all nodes have to have the template available in the same place (although it could be useful to vary the template contents according to gear profile):
    # mkdir -p /etc/openshift/templates
    # git clone template /etc/openshift/templates/jbossews2.git
  4. Now modify the broker to specify this as the default. In /etc/openshift/broker.conf:
    DEFAULT_APP_TEMPLATES=jbossews-2.0|file:///etc/openshift/templates/jbossews2.git

    … and restart the broker:

    $ service openshift-broker restart

    Of course, with multiple brokers, you need to do this for all of them.

At this point, whenever you create a new jbossews-2.0 application, it will default to using your template and the changed access log format.

Advertisements

Highly available apps on OpenShift

One question we’re working through in OpenShift is how to make sure applications are highly available in the case of node host failure. The current implementation isn’t satisfactory because a single gear relies on its node host to function. Host goes down, gear goes down, app goes down.

We have scaled applications which expand the application out to multiple gears, but they have a single point of failure in the proxy layer (all requests go through one proxy gear). If there is a database cartridge to the app, that also is a single point of failure (we don’t offer database scaling yet). Finally, there’s no way to ensure that the gears don’t actually end up all on the same node host (except by administratively moving them). They are placed more or less randomly.

This is a hot topic of design debate internally, so look for a long-term solution to show up at some point. (Look for something to crystalize here.) What I want to talk about is: what can we do now?

If you have your own installation of OpenShift Origin or OpenShift Enterprise, here is one approach that may work for you.

  1.  Define a gear profile (or multiple) for the purpose of ensuring node host separation. It need not have different resource parameters, just a different name. Put the node(s) with this profile somewhere separate from the other nodes – a different rack, a different room, a different data center, a different Amazon EC2 region; whatever will satisfy your level of confidence criteria in what size failure you can expect your app to survive.
  2. When you create your app, do so twice: one for each gear profile. Here I’m supposing you’ve defined a gear profile “hagear” in addition to the default gear profile.
    $ rhc app create criticalApp python
    $ rhc app create criticalAppHA python -g hagear

    You can make them scaled apps if you want, but that’s a capacity concern, not HA.

  3. Now, develop and deploy your application. When you created “criticalApp” rhc cloned its git repository into the criticalApp directory. Code up your application there, commit it, and deploy with your normal git workflow. This puts your application live on the default gear size.
  4. Copy your git repository over to your HA gear application. This is a git operation and you can choose from a few methods, but I would just add the git remote to your first repository and push it straight from there:
    $ rhc app show criticalAppHA

    Output will include a line like:

    Git URL = ssh://3415c...@criticalAppHA-demo.example.com/~/git/criticalAppHA.git/
    

    … which you can just add as a remote and push to:

    $ cd criticalApp
    $ git add remote ha ssh://...
    $ git push ha master

    Now you have deployed the same application to a separate node with profile “hagear” and a different name.

  5. Load balance the two applications. We don’t have anything to enable this in OpenShift itself, but surely if you’re interested in HA you already have an industrial strength load balancer and you can add an application URL into it and balance between the two backend apps (in this example they would be http://criticalAppHA-demo.example.com/ and http://criticalApp-demo.example.com/). If not, Red Hat has some suitable products to do the job.

This should work just fine for some cases. Let me also discuss what it doesn’t address:

  • Shared storage/state. If you have a database or other storage as part of your application, there’s nothing here to keep them in sync between the multiple apps. We don’t have any way that I know of to have active/active or hot standby for database gears. If you have this requirement, you would have to host the DB separately from OpenShift and make it HA yourself.
  • Partial failures where the load balancer can’t detect that one of the applications isn’t really working, e.g. if one application is returning 404 for everything – you would have to define your own monitoring criteria and infrastructure for determining that each app is “really” available (though the LB likely has relevant capabilities).
  • Keeping the applications synchronized – if you push out a new version to one and forget the other, they could be out of sync. You could actually define a git hook for your origin gear git repo that automatically forwards changes to the ha gear(s), but I will leave that as an exercise for the reader.

It might be worth mentioning that you don’t strictly need a separate gear profile in order to separate the nodes your gears land on. You could manually move them (oo-admin-move) or just recreate them until they land on sufficiently separate nodes (this would even work with the OpenShift Online service). But that would be somewhat unreliable as administrators could easily move your gears to the same node later and you wouldn’t notice the lack of redundancy until there was a failure. So, separating by profile is the workaround I would recommend until we have a proper solution.

Checking out Hibernate with STS

I have cause to revisit my post about importing a git checkout into Eclipse. I need to take a look at the Hibernate code, which is on github at git://github.com/hibernate/hibernate-orm.git, and given the somewhat convoluted nature of this code, I need an IDE to help navigate it.

Now, I hear that with Eclipse Indigo (3.7), which is included with the latest STS 2.9 (which is what I use), the EGit plugin is included out of the box (which, for the purposes of this post, is what I’m doing – completely stock install). That’s helpful. See, previously, if you wanted to do something with git, you would find no evidence within Eclipse that it could. If you figured “there must be an extension for that” and searched for “git” from the extensions wizard, there would be no hits. Because what you needed to look for was “JGit” or “EGit” – you big dummy. An example of what I consider low discoverability that’s pervasive in Eclipse/STS. But I digress.

At least EGit has had a couple years to bake since my post. I went to File->Import->Git->Projects from Git and put in the URI above. This seems pretty straightforward:

Image

I’m not sure why it parses the URI into Host and Repository path here. Is there some reason you’d want to customize these?

In the next step, I pick the branches from the repo I want and proceed to the “local destination” dialog.

Image

These steps might be somewhat confusing to those who don’t know git and just want to take a look at some code. Since git is distributed, you don’t just get a point-in-time checkout from a repo, you get your own copy of the repo – or as much of it as you want. Basically it’s asking where I want my copy of the repository and checkout to go. The checkout (“initial branch” here) will go in the directory, and the repo will go in a .git subdirectory. “origin” is the name given to the repository I cloned this from, in case I want to sync with that later. That might be kind of obvious to someone familiar with git, but how about some tips for those who aren’t?

My question: why doesn’t this all simply default to being inside the workspace? What’s a workspace for, if not the project contents? As you can see, the default is to create a ~/git directory and checkout/clone the repo there.

Next step, three inscrutable options for how to deal with the resulting project(s) that have been checked out:

Image

OK. These seriously need some explanation. What do these do?

“Import existing projects” gets me nowhere in this case, as it requires Eclipse project descriptors to be included in the checkout, and they’re not. Arguably, they probably shouldn’t be. I just get the error “no projects found” if I try this. But that means I need to figure out myself how to get Eclipse/STS to interpret this checkout properly.

“Use the New Project wizard” is an option I don’t really understand. It just dumps you into the new project wizard you would get by clicking the new project button (generally the first button in the toolbar). This is also where you end up if you click “Finish” instead of “Next” anywhere along the way. I guess I could make use of the directory just created. I  also can’t go “back” and choose another option from here; cancel, and I’m back to square one. In general, I find the “New Project wizard” one of the most confusing things about Eclipse/STS, because there are so many options, many sounding similar yet meaning something completely different, and no explanations of what you can expect to get. Do I really have to go looking for doc that should be a click away? I digress.

“Import as general project” basically just creates a project with the given content and no organization. STS recognizes the different file types, of course, but there’s no concept of where the classpaths begin, how to build and test the project, anything like that – just plain directories with content. This won’t get me to my goal, which is to be able to look up class hierarchies, implementors of interfaces, etc. However, having done this, I can try to configure the project to get it to where STS understands these things.

I’m interested in the 3.6 branch of Hibernate, which is a Maven project (you can tell from the pom.xml – woe betide you in the Java world if you don’t recognize Maven when you see it. The “master” branch seems to be using Gradle). So I can right-click the project and Configure -> Convert to Maven Project.

By the way, let me point out something that didn’t work at all: creating a new project with the wizard “Maven -> Checkout Maven Projects from SCM”.

Image

This is apparently not aware of the EGit plugin, because there’s no SCM protocol listed here (the first box  is greyed out). If I click “Finish” here nothing happens except the dialog exits. I think it would probably work if I added a m2e SCM connector like the link suggests, but how would I know to do that?

Alright, so now I have a Maven project. Right away in the top-level pom.xml I get a “Project build error: Unresolveable build extension: Plugin org.jboss.maven.plugins:maven-jdocbook-style-plugin:2.0.0 or one of its dependencies could not be resolved: Could not find artifact org.jboss.maven.plugins:maven-jdocbook-style-plugin:jar:2.0.0 in central (http://repo1.maven.org/maven2)”. I happen to know what this is about because I know there are a bunch of JBoss dependencies not in Maven Central. How would I know that if I didn’t know? Google, I guess. Fortunately searching for that exact error message gets me right to a StackOverflow question about exactly the same thing, which someone has helpfully solved. I love SO, I just hate that it has to exist. Documentation is always about how to use something the right way, not what to do when something goes wrong. SO fills that gap.

So, add the repository information to the pom.xml – or, better, to my Maven settings.xml (which I had to create since STS is providing Maven in this setup) and on to the next problem. Two, actually (always seems to be the way of it – removing a build problem just uncovers more). These are related to “Missing artifact commons-logging”. A little Google sauce on that turns up this blog post (like the name, kinda like my blog!) about the death of the commons-logging dependency. Gotta love trying to support these old builds from a public ever-changing repo. Problem is, the Hibernate pom (actually the parent pom, which is in a subdirectory! huh?) uses the hack from that article, but the repo supplying the dummy dependencies seems to be down. So perhaps I should try the exclusions suggested by commentors in that blog? I found something that looks handy: in the pom dependency hierarchy, right-click and choose “Exclude Maven artifact”:

Image

Sadly, this doesn’t work:

Image

But here’s another StackOverflow suggestion. This seems to work, after removing the existing commons-logging dependencies and adding those ones in the parent pom, and (finally) right-clicking on the project, Maven -> Update project configuration. The errors are gone, and (I suspect) so is all the Maven-fu I can expect today.

Unfortunately I’m still not at my goal – I just have the Maven nature working now.

Turns out, this wasn’t quite the right path. What I’m looking at here are multiple Maven projects, with interdependencies. There’s no way I’m ever going to get anything useful from this in a single STS project. What I need to do is import these as multiple projects. In the meantime, delete the existing project (but leave the checkout) so it doesn’t get in the way.

So here’s what I do: File -> Import -> Existing Maven Projects and enter the path to my local checkout as the “Root Directory”:

If I select all the projects, they’ll all be created as interdependent workspace projects, each with build path and so forth configured according to Maven.

With lots of errors, of course… thousands, in fact. But let me start with the Maven problems, which are probably the source of the rest. Looks like all of the Maven errors are of the form “Plugin execution not covered by lifecycle configuration: org.jboss.maven.plugins:maven-injection-plugin:1.0.2:bytecode (execution: default, phase: compile)” – with a different plugin each time. I remember the import screen warned about some problems that would need to be resolved later – this seems to be what it was talking about.

Well, much later now, I think the Maven errors were mostly irrelevant. Those were due to the change to the m2eclipse plugin which broke the world for a lot of Maven users in Eclipse. Most of them were things that looked like it was safe to have m2eclipse “ignore” as recommended there. I went ahead and ran some of the goals that looked important (antrun:run and injection:bytecode in hibernate-entitymanager, the latter in hibernate-core) from the command line. Not sure they made much difference. I did Maven -> Update Project Configuration on everything changed and most of the red X’s went away.

I also ran into this problem and crashed a few times just by mousing over the “Window->Browser” menu before adding “-Dorg.eclipse.swt.browser.DefaultType=mozilla” to my STS.ini to avoid it.

At this point, the only problem seems to be that hibernate-entity has a ton of tests with imports like this:

import org.hibernate.ejb.metamodel.Customer_;
import org.hibernate.ejb.metamodel.Order;
import org.hibernate.ejb.metamodel.Order_;

… and then goes on to use these classes with underscores, which aren’t there. Evidently they’re supposed to be generated at some point, but I’m not sure how. I don’t really care about running these tests, just wanted to look at the framework code, so although STS reports 14382 Java problems, I can consider my work done here. Boy, that was easy!

One more note: I went back and added the git SCM connector for m2eclipse to try it out. It worked… but poorly. The way that worked was to select “git” for the scheme, then put in the git:// URI for the project, then wait for a popup to select the projects to import. If I reversed order or didn’t wait, I got either an error or nothing after hitting “Finish”. Hmm… hope that’s better in the next update. And, interestingly… the checkout/repo went into the workspace.

git rm; application toast pool; G1 bug

Alright, I know I saw it somewhere. How do I get git to add deletion of files to the transaction (when I deleted them w/out git rm and don’t want to git rm the whole list of them)? Can’t find the original answer but Stack Overflow’s top answer works for what I want. When I’ve done all my other adds and find there’s a bunch of deleted files left over… “git add -u”

On to fun stuff. I thought it seemed a bit wasteful to create toasts over and over, so I created a toast pool.

    // repository for created toasts
    private Map<Long,Toast> mToasts = new HashMap<Long,Toast>();
    private static final Long TOAST_LOG_DELETED = new Long(R.string.log_entry_deleted);
    private static final Long TOAST_LOG_CREATED = new Long(R.string.new_log_entry);
    protected void showToast(Long id) {
        Toast t = mToasts.get(id);
        if(t==null)
            mToasts.put(id, (t = Toast.makeText(this, id.intValue(), Toast.LENGTH_SHORT)));
        t.show();
    }

Initially I put this in an Activity that was creating a lot of toasts… then I thought, why not do this application-wide? Then if I need to show a toast with an Activity that’s finish()ing, it’ll still show. Plus better to have just one implementation and pool. So I put it in my Application object (WhenDidI) and the constants in my constant class (C). Now my toast calls look like this:

    ((WhenDidI) getApplication()).showToast(C.TOAST_LOG_DELETED);

A little wordy, but otherwise – good idea, no?

Asked my first question on StackOverflow yesterday. Seeing a bug with TimePicker apparently only on my G1. I’ll probably figure something out…

Most things in WhenDidI work as expected at this point, so I’m going to start getting familiar with styles and applying them everywhere. About time. This is a great post to help with understanding how styles are used in Android.

Importing git checkout project into eclipse workspace

Note: for some reason this post still gets a lot of hits. My post much later might be more helpful.

Trying to figure out how best to import Android projects into eclipse, particularly from a version control checkout. I’ve had the same trouble with trying to use samples downloaded from book/tutorial sites. What I’m trying to do:

  • I want to locate the project in my eclipse workspace.
  • I don’t want eclipse-specific files in version control – those seem to cause problems when moving between different eclipse installations. Or, in the case of downloaded examples, these files aren’t provided.
  • Use git; other VCSs have Eclipse plugins to simplify this – git does too, but it seemed half-baked and I shouldn’t have to use a plugin to do this.

Problems:

  • If I check out the project outside my workspace, and then try to create a project in eclipse using that as source, eclipse does not move or copy it into the workspace; it uses the original location (which makes sense)
  • If I check out the project within my workspace, and then try to create a project using it as source, eclipse refuses because of the location.
  • If I use File -> Import -> General -> Existing projects into workspace, eclipse expects to see its project descriptors (that I deliberately excluded with .gitignore) and refuses the import.
  • No other options seem appropriate to doing this

Here’s one way to do it:

  1. Check out the project somewhere.
  2. Use the “create project” dialog to create an Android project using this location as source. This adds the requisite eclipse project descriptor files.
  3. Close the project and delete it (leaving the files in place of course).
  4. Move the project directory into the eclipse workspace.
  5. Use File -> Import -> General -> Existing projects to import the project directory – since the project files are there now, this works fine.

I wonder if I’m just nutty leaving the eclipse project files out of the checked-in version. But they just don’t seem to belong there, neh?