Response to “PaaS for Realists” re OpenShift

I work on OpenShift but would not claim to be the voice of OpenShift or Red Hat. I came across Paas for Realists the other day and thought it could use a quick response. No one is vetting it but me :)

There are some good points in this article. Just like other technology, a PaaS does not magically solve anything. Actually it carries forward all of the flaws of the technology it includes, and then adds a layer of complexity on top. The PaaS does a lot of work for you, and that can be really nice, but it can get in the way too, and the operational model is not always a good fit. It’s important to know what you’re getting and what you’re giving up by employing such a solution.

I wish I could just annotate the article, but quote and response will have to do…

Magical autoscaling

As I said in my previous post, this really doesn’t exist. Your application has to be DESIGNED to scale this way.

Agreed; auto-scaling is a difficult problem to tackle, and if your application isn’t designed for stateless, horizontal scaling up and down, it’s just not going to work well. This isn’t really specific to PaaS.

I would note, by the way, that while OpenShift does enable auto-scaling web frameworks according to a default algorithm, you can either override the algorithm or just disable it and scale manually, whatever works best for your app. One size does not fit all. Sticky sessions are built in if you need that, though sessions are only clustered with the JBoss EAP cartridge (so for all others, in-memory or on-disk sessions will be lost if you scale down or lose a gear).

Magical Autorecover

Just like autoscaling, this is also not what you think it is. Unless your application maintains exactly ZERO state, then you will never see this benefit.

OpenShift doesn’t really claim magical autorecovery or self-healing (yet). Agreed, if you’re storing state on a gear, it can be lost to an outage. Scaling just means you have copies of your webapp. If you want to maintain state in a production setting, you’ll need to store it off-PaaS. You would need to set up something to account for outages in a more traditional deployment too; OpenShift just doesn’t do anything special to help (yet).

I’m sure someone will call me on this and I’m willing to listen but I do know for a fact that the autofailover model of things like your MySQL instance depend on migratable or shared storage (at least from my reading of the docs).

Databases can generally be made HA via replication of some variety. We’ve done some R&D on making database cartridges scalable in order to provide HA storage to an app; it will probably happen at some point – definitely a weakness right now. For now, if you want a HA database, set it up outside the PaaS. You would have had to do that without a PaaS anyway, and your DBAs are already pretty good at it, right? What OpenShift *does* get you is the ability to develop your app on the PaaS against a throwaway DB, then when you want to go to production on the PaaS, you push exactly the same code to your production app and just set some environment variables to point to the production DB.

Same thing for storage; if you want HA storage, set it up outside the PaaS. This is even trickier to solve in-PaaS than DBs, but we’re hoping to address it based on the version of NFS that just came out with RHEL 7.

Also one of the more hilarious bits I’ve found is the situation with DNS. I can’t count the number of shops where DNS changes where things like wildcard DNS were verboten. Good luck with the PaaS dyndns model!

OpenShift doesn’t need wildcard DNS, but it does use DDNS, and that can definitely be a sticking point to demo in organizations where even delegating a subdomain is a bureaucratic battle. But at least it’s a battle you only have to have once at solution deployment, instead of for every single app you deploy. Do you have a better suggestion for how to dynamically make apps available? Most places don’t like to provide a pool of IPs to burn per-app; even more than they don’t like DDNS.

Operational Immaturity

Any tool that uses MongoDB as its persistent datastore is a tool that is not worth even getting started with. You can call me out on this. You can tell me I have an irrational dislike of MongoDB.

You have an irrational dislike of MongoDB :)

Well alright, not totally irrational. The default write concern for earlier versions of the mongo clients left something to be desired if you cared about DB integrity, we’ve encountered memory leaks when using SSL connections, and we haven’t made the leap to MongoDB 2.6 yet. I’m sure you have more horror stories to share.

But the fact is, we’ve been running using MongoDB as the core of OpenShift for years now – both in the public service and for our private solution – and it has seriously been very solid. Our ops guys said (and I quote) “We had a couple of outages early on that were traced back to mongo bugs but generally we don’t even think about it.  Mongo just keeps ticking and that’s fantastic.”

Was the use of MongoDB your only criticism here? I think we do provide pretty thorough instructions on how to make the OpenShift infrastructure solid.

Additionally I’ve found next to zero documentation on how a seasoned professional (say a MySQL expert) is expected to tune the provisioned MySQL services. The best I can gather is that you are largely stuck with what the PaaS software ships in its service catalog. In the case of OpenShift you’re generally stuck with whatever ships with RHEL.

Tuning is a definite weakness. It is hard to both provide software setup you don’t have to think much about, and also allow you to administer it. This is a known concern that I think we will work toward with the docker-based next generation.

You’re not stuck with whatever ships with RHEL on OpenShift. Publicly and privately we’ve added support for several SCLs which can provide completely new platforms or just different versions than what ship with RHEL. You can also add a custom cartridge to run pretty much any technology that speaks HTTP (and depending on your needs, many that don’t).

Another sign of operational immaturity I noticed in OpenShift is that for pushing a new catalog item you actually have to RESTART a service before it’s available.

Do you mean, to add a cartridge, you need to restart something? If so, that’s not really true, although it was in the past, and I’m not sure we’ve clarified the directions well enough since.

Disaster Recovery

After going over all the documentation for both tools and even throwing out some questions on twitter, disaster recovery in both tools basically boils down to another round of “good luck; have fun”.


Again based on the research I’ve done (which isn’t 1000% exhaustive to be fair), I found zero documentation about how the administrator of the PaaS would back up all the data locked away in that PaaS from a unified central place.

The OpenShift infrastructure just depends on redundancy for HA/DR. Hopefully that’s pretty straightforward given the directions.

To make your applications recoverable, use highly-available storage for the nodes and back it up. There’s not a great deal of detail in that section of docs, but does there need to be?


Affinity issues make the DR scenario even MORE scary. I have no way of saying “don’t run the MySQL database on the same node as my application”.

Well, that’s true, but with OpenShift you *do* have the ability to define availability zones and the gear placement algorithm will ensure that your scaled app is spread across them. Once we get scaled DB cartridges I expect the same will apply for them (see above re “in-PaaS DBs aren’t for production yet”). And if that’s not good enough for you, we have a hook to customize the gear placement algorithm until it is good enough for you.

Unless your engineering organization is willing to step up to the shared responsibility inherent in a PaaS, then you definitely aren’t ready. Until then, your time and money is better spent optimizing and standardzing your development workflow and operational tooling to build your own psuedo-PaaS.

Agreed, your developers are not going to just be able to ignore that they’re working and deploying on a PaaS. It’s not a magical solution. It’s a specific way to provide services with pros and cons all its own and that context needs to be understood by all stakeholders.

It *may* be possible to create your own PaaS specific to your needs and be happier with it than you would be purchasing someone else’s solution. But I will say that we have run into a lot of forward-thinking companies that did exactly this within the last few years, and now are desperate to get away from maintaining that solution. Keeping up with ever-churning platforms and security updates always takes more work than anyone expects. So if you think PaaS is right for you, also ask yourself: do you want to be in the building-a-PaaS business, or the using-a-PaaS business?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: