Exploring 3 – docker

More unreliable ruminations -

When Docker started to make a splash, I took a quick look at it, you know, the basic tutorial. All very nice, but not too much depth. And even though the rest of the OpenShift team has pivoted to this platform fairly quickly, I’ve been waiting until I would actually have some real time to devote to it before digging in deeper.

Although I know that at the pace this stuff is moving, RHEL 7 is already far behind, I brought up a RHEL 7 host and started running through https://access.redhat.com/articles/881893 which has a little more meat to it as far as introducing Docker capabilities. Under RHEL 7, Docker is in the “extras” channel (and relies on one pkg in the “optional” channel). It’s useful to know that the “extras” channel is actually supported (unlike “optional”), but not on the same terms as the rest of RHEL – things in this channel are allowed to move quickly and break compatibility. That’s a good place for Docker, since I know our team is still collaborating heavily with Docker to get in features needed for OpenShift. I expect there will be a sizeable update for RHEL 7.1, although chances are we’ll be using Atomic anyway.

Atomic ships tmux but not screen. I guess it’s time for me to finally make the leap. As tempting as it is to just remap the meta key to C-a, I should probably get used to using the defaults.

The first thing that would probably help me to understand Docker is an analogy with Docker registries/repositories and git. Docker is clearly informed by git and VCS, using some of the same verbs (pull, push, commit, tag) but assigning different semantics.

This article clarified the similarities and differences in terms (although it’s not clear when it was written, looks like about a year ago… seriously, an undated blog post on new technology? How does this keep happening?). Dockerhub is approximately like Github… repositories are approximately like Github repos. The location of the image layer information doesn’t seem to be the same for me, but I don’t know if that’s because Docker changed in the meantime or because it is packaged differently for RHEL/Atomic.

docker pull

So, you “docker pull” an image. It’s a little confusing where you’re pulling it from and to. “To” turns out to be clearest… a local cache, which nothing ever tells you where that is, but it looks like on RHEL 7 it’s under /var/lib/docker/ – there’s image metadata at /var/lib/docker/graph/ and perhaps some actual content at /var/lib/docker/devicemapper/ but I’m having trouble seeing exactly how the image data is stored – I’m sure this is confusing for a reason. Open question for now.

Here’s a handy alias:

# alias json="python -mjson.tool <"

Now you can pretty-print json without having to think much about it:

json /var/lib/docker/graph/216c11b99bd09033054595d08c28cf27dabcc1b18c2cd0991fce6b1ff1c0086f/json | less

Docker storage is configurable in /etc/sysconfig/docker-storage and under Atomic, perhaps predictably, it is customized to live under /dev/atomicos/. Though there’s still plenty under /var/lib/docker.

So this is a bit like a system-wide git repository. You can contact as many “remotes” (registries) as you like, and pull down “branches” (images) composed of successive “commits” (layers) potentially with “tags” (tags! although tags do double duty as points in time and moving like branches). Once they’re present locally you can fire them up, modify them (with another commit) and push the results back to a registry.

It’s less than crystal clear to me how “docker pull” chooses a remote, i.e. how registries are determined. OK, if you “docker pull registry.access.redhat.com/rhel” it should be apparent where that’s coming from. But despite the docker hub reportedly being disabled, if I “docker pull ubuntu” or “docker pull nginx” those load up just fine – from where? Evidently Docker Hub isn’t disabled. Here’s how it seems to work:

docker pull <word e.g. "ubuntu">  = get images from public "word" repository on Docker Hub
docker pull <word>/<repo> = get images from repo owned by <word> account on Docker hub
docker pull <hostname or IP>/<repo> = get images from repo on other registry

In all cases, you can add a :tag to pull only a specific tag (and any images it is based on) rather than all of the tags in the repository.

As with git repos, you have a local concept of the remote repo which can be out of sync. So you have to push and pull to sync them up as needed.

docker commit / build / tag

If you run an image as a container, you can then commit the result as an image. If you commit it with the same name as an existing repository, it’s implicitly tagged as :latest.

Similarly you can use “docker build” with a Dockerfile that specifies base image and commands to run against it, then commit the result as an image in a repository.

Finally, you can just re-tag any image in the local cache with any repository and tag you want (within the bounds of syntax, which are pretty loose). So “docker tag” doesn’t just apply tags (and moving tags = branches) but also repositories.

docker push

Having created an image in a repo, docker push is the reverse of docker pull… and the repo indicates where it will go.

You can’t docker push to one of the root repos (like just plain “mongodb”). You can of course pull that, re-tag it with your own Docker Hub id (e.g. “docker tag mongodb sosiouxme/mongodb”) and then push it (assuming you’ve logged in and want it on your Docker Hub account).

Finally if you have tagged your image with a repo name that includes hostname/IP, then docker push will try to push it to a registry at that hostname/IP (assuming it exists and you have access). RHEL 7 ships docker-registry, but Atomic does not at this point – and why should it when you can just run the registry itself in a container?

Exploring 2 – journal

I have been reading through Lennart Poettering’s ever expanding up to the seventeenth installment of his ongoing series on systemd for Administrators without much to say here. Good stuff.

Number 17 is about the journal, which is basically a replacement for syslog. This answers my earlier question of how systemd displayed the log lines from httpd… the journal is hooked up by systemd to capture syslog and kernel log entries as well as stdout/stderr for any processes it manages. What I saw in the httpd status output there would be the stdout from starting httpd… the journal isn’t following the actual log files created by httpd (you’d need to configure httpd to log messages to syslog or journal).

The journal is really cool, though. It natively solves a lot of annoying things about system logs, mainly by attaching a ton of metadata to log entries, including automatic and unfakeable items like cgroup, pid, and executable. And then indexes by that and presents a nice filtering interface with the journalctl client (incidentally allowing users to access their own log entries). If we had this in OpenShift 2, we wouldn’t have needed a plugin for rsyslog7 to add these kinds of attributes to gear syslogs, and gears would not have needed to store their own logs at all since they could just access their own journal entries from the host with journalctl (although… I would need to check how access is controlled; if it’s by UID and not SELinux context then we would need to do something special because UIDs can be reused under OpenShift). I bet we’ll use this for v3.

One thing to note under RHEL 7… the default install doesn’t enable the persistent journal – all you have is whatever is stored in /run/log/journal since the last boot. However, it’s easy to enable the persistent journal by just creating /var/log/journal. At this point you can nuke rsyslog and just let the journal capture everything. Also, bash tab completion doesn’t seem to be set up for journalctl attributes as indicated in the blog (there’s probably a simple way to enable that too).

Exploring 1 – systemd

Starting to poke around systemd a bit more.The following is some running commentary – I may have no idea what I’m talking about.

I have to admit to myself I didn’t truly understand sysvinit all that well, I just knew how to get done what I needed to. Other than being used to typing “service xyz start” and “chkconfig xyz on” and looking in /etc/init.d for control scripts and /etc/rc*.d for symlinks, I couldn’t have answered a lot of specific questions about how it works. So really, aside from having to update my muscle memory for those things, I don’t have a huge attachment to sysvinit.

Google directed me to what looks like a good series of blog posts on systemd starting at http://0pointer.net/blog/projects/systemd-for-admins-1.html – starting in 2011 so there’s probably been a lot of drift since, but still looks like a good starting point with motivation and technical underpinnings. There’s just always more to learn about the OS.

systemctl status has a great deal more info than service status ever did, because systemd seems to standardize a bunch of stuff that was just left up to the control script before. Consider for httpd:

# service httpd status
httpd (pid 7114) is running...

So, we get that there is a running process, because the httpd daemon put down a pidfile and the process with that pid is running and looks like an httpd process. There are several other variations, including not being running, or having a pidfile but no corresponding process… well, the httpd service script could have put just about anything in its status output, tracking down log files and process trees would just be a little extra work there. systemd seems to keep track of a bunch more:

# systemctl status httpd
httpd.service - The Apache HTTP Server
 Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
 Active: active (running) since Tue 2014-12-02 08:11:22 EST; 12min ago
 Main PID: 23773 (httpd)
 Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
 CGroup: /system.slice/httpd.service
 ├─23773 /usr/sbin/httpd -DFOREGROUND
 ├─23774 /usr/sbin/httpd -DFOREGROUND
 ├─23775 /usr/sbin/httpd -DFOREGROUND
 ├─23776 /usr/sbin/httpd -DFOREGROUND
 ├─23777 /usr/sbin/httpd -DFOREGROUND
 └─23778 /usr/sbin/httpd -DFOREGROUND
Dec 02 08:11:22 lmeyer-1201-rhel7 systemd[1]: Starting The Apache HTTP Server...
Dec 02 08:11:22 lmeyer-1201-rhel7 systemd[1]: Started The Apache HTTP Server.

We get a pointer to the systemd unit file for httpd, whether it’s enabled (for running automatically at boot), whether it’s running (“active”) and for how long, process tree (which we can obtain with confidence because the daemon is put in a cgroup at start), some log entries, and some service-specific “Status” about traffic. That’s pretty handy, and not to say that the service script couldn’t have done all this, but systemd seems to standardize it.

# cat /usr/lib/systemd/system/httpd.service
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

I looked in the unit file and I’m curious how it does all this, since there’s no entry for logs or status. Something to look out for.

Part 2 gets into the usage of cgroups, which seems like a neat use case. I notice that the cgroup names seem to be longer than when the article was written, so it’s helpful to expand the column in the ps command suggested for viewing processes with cgroups:

# alias psc='ps xawf -eo pid,user:16,cgroup:64,args' 
# psc
23773 root   1:name=systemd:/system.slice/httpd.service /usr/sbin/httpd -DFOREGROUND
23774 apache 1:name=systemd:/system.slice/httpd.service \_ /usr/sbin/httpd -DFOREGROUND
23775 apache 1:name=systemd:/system.slice/httpd.service \_ /usr/sbin/httpd -DFOREGROUND
23776 apache 1:name=systemd:/system.slice/httpd.service \_ /usr/sbin/httpd -DFOREGROUND
23777 apache 1:name=systemd:/system.slice/httpd.service \_ /usr/sbin/httpd -DFOREGROUND
23778 apache 1:name=systemd:/system.slice/httpd.service \_ /usr/sbin/httpd -DFOREGROUND

I read through http://0pointer.de/blog/projects/systemd.html which is a long and detailed introduction to the motivations behind systemd. Probably should have done that first.

Back to the future (of OpenShift)

I started this blog originally for just sort of writing down random stuff I tried or discovered. It morphed over time into very rare posts along the lines of “I just spent a week figuring this out, let me write it down to save everyone else the trouble”.

Well, OpenShift Enterprise 2.2 is out the door, and that will be in maintenance mode while we work on version 3. Just when I felt like I knew something about v2, it’s time to return to being a dunce because the world has been upended for v3. So maybe it’s time to return to stumbling around and writing down what happens.

Everything old is new ag…. no, wait:

Everything new is really, really new

Approximately nothing from OpenShift v2 will survive recognizably in v3. It will be as different as systemd is from sysv, as different as Linux is from Windows, as different as solar energy is from the Hoover dam. Here’s what’s on my hit list to get up to speed on (let me know what I missed):

RHEL 7 / Atomic

OSE 2 runs on RHEL 6. About the time Fedora 20 and Ruby on Rails 4 came out, it became evident that trying to make it span RHEL 6 and newer platforms was going to be way more trouble than we wanted. We gave up on that and left Origin users to run on CentOS 6 rather than try to keep including Fedora.

This brings some good things, though. Managing dependencies for OSE 2 on RHEL 6 has been a bit of a nightmare. All signs point to that going away completely for v3. As in, you might not even need yum at all. If the eventual platform we recommend is Atomic, platform updates will be whole-system run via rpm-ostree (AKA atomic). If so, then I’ll need to know about that distribution mechanism. If not, it still looks like there will be a lot less to install and configure on the actual OS.


  • rpm-ostree / atomic
  • systemd – have to understand more than just “systemctl enable foo; systemctl start foo” – how to define services, how daemons are spun off and monitored, where logs go…
  • firewalld – is this just a frontend to iptables?
  • btrfs?


go is the new hotness. Ruby on Rails is old and broken. OK, not old and broken, but docker, kubernetes, etcd, and the OpenShift layer on top are all go-based. Fortunately I used C all through college… picking up go doesn’t look difficult, should be fun.

golang vs gcc-go – the former is what most are using, the latter gets us more supported platforms if it works with the codebase.


Docker will be replacing our homegrown containers. It’s a formalization of a lot of the same concepts – creating and containing processes with regards to network, file access, resource usage, etc. Some questions for me to get through:

  • How do I get files into/out of a container? Bind mounts, other kinds of mounts, …? What happens when it goes away?
  • What exactly happens with exposing container networking?
  • How does SELinux contain a Docker container?
  • How do cgroups contain a Docker container in RAM/CPU/etc?
  • How do I control what user runs the processes in a container?
  • How does UnionFS compose multiple containers?
  • How do I configure where images come from?
  • How do I figure out what went wrong after one exits and goes away?

… and a million other things.


Kubernetes is one orchestration layer on top of Docker. It will handle things like ensuring you have the expected number of copies of an image running across the various hosts on the cluster, and providing a proxy (aka “service”) for reaching them at a stable location.

Kubernetes introduces the concept of “pods” which are essentially just related containers running together on a host and sharing resources. As far as OpenShift is concerned, pods will likely only ever have a single container and thus be synonymous, but the terminology is there nonetheless. Do not confuse “pods” with “apps” (which are also composed of containers, but potentially spread across multiple hosts).

Things to learn:

  • Kubernetes masters present a REST API, so need to know that a bit.
  • How are multiple kubernetes masters synchronized? Just via entries in etcd, or more directly?
  • How do kubernetes masters communicate with minions (kubelets)?
  • How do services/replication masters determine whether a container/pod is working or not?


Distributed key-value store. I’m not sure why we needed another one, but it seems that it’s going to be the store for lots of critical stuff. Which critical stuff? Good question, probably not *all* of it… What else might we use for a data store?

Aside from the general capabilities of etcd, I need to learn how to cluster and shard them, and how the RAFT consensus synchronizer works (or when it doesn’t work).

OpenShift v3

Of course, this is going to add a further layer on top of Kubernetes, a layer to define apps and user access to them. A lot of it is still in pretty early stages, e.g. there’s not really any concept of users or access controls yet. That’ll change.

  • REST API (parallel but separate from Kubernetes)
  • Building container layers from source code
  • Deployment strategies
  • How does OpenShift influence the placement algorithm with parallels for the scaled/HA apps, zones, and regions of v2?
  • What does the routing layer look like? (We aren’t simply going to expose Kubernetes services) Good gracious, the networking looks to be complicated for this.
  • How will we define and mount storage for use by containers / apps?

Angular.js web console

Having a web application server is so last year (or maybe decade?). The data is all available from REST APIs… now your web app can just be static pages with a ton of JavaScript doing all the work on the client side. This replaces the OpenShift v2 web console app. At least it’s one less service to keep running, and you won’t need to hit “reload” all the time to watch things changing.

Is anything staying the same?

Technology-wise, nothing is staying the same. Get ready for that (I’ve marveled that the rest of the team could pivot so quickly). But we’ve spent a few years now building a PaaS, and of course there are certain patterns that are going to pop up no matter what technology we use. Despite all the technology changes, those same issues are probably what we’ll be beating our heads against, and where hopefully our previous experience will help OpenShift prevail.

Infrastructure, nodes, and routing

OpenShift will probably constitute the infrastructure only – the apps will actually run on hosts that run Linux, Docker, and kubelets. But the general pattern will remain – an orchestration interface, a cluster of compute nodes, and routing layer to reach them.

Composing apps

Apps will still be put together from several components – potentially several containers (I don’t think we’ll call them gears), each potentially composed of some kind of framework plus some of your code. Defining and wiring these together will be the core of what OpenShift continues to do.

Access control

We’re still going to have users. There will still be teams. There may be more layers (e.g. probably admins, “utility” users). It will still be necessary to define things that those users and teams can access. And it will still be necessary to interface with the various ways in which enterprises already define users, groups, authentication, and authorization (Kerberos, LDAP, …).

Proxies (AKA layers of indirection)

In OpenShift v2, there are a number of ways in which your request to an app can actually reach the thing that answers the request, often going through multiple proxies. In perhaps the most complicated case, with an HA app setup, you lookup the app by name (DNS itself consists of several layers of indirection) and reach the external routing layer, which forwards to a node host routing layer, which forwards to a load-balancing gear layer, which forwards to another node’s port proxy, which finally forwards to the application server running in a gear. V3 will differ in the details, but not the pattern.

These proxies don’t exist just to peel back layers of the onion; each point provides an opportunity to hide complicated behavior behind a simple facade. For example:

  • DNS records provide all sorts of routing opportunities, including directing the user to a data center that’s available and geographically close to them.
  • A routing layer can monitor multiple endpoints of the application such that outages are detected and requests are directed to functioning endpoints. These can also hide the fact that gears are being moved, rolling deployments, etc.
  • The node host routing layer can hide the fact that a gear was actually idle when a request arrived, bringing it up only when needed and conserving resources otherwise.
  • The load-balancing gear layer balances traffic and implements sticky sessions.

As you can see, proxies are actually where a lot of the “magic” of a PaaS happens, and you can expect this pattern to continue in v3.

Implementing an OpenShift Enterprise routing layer for HA applications

My previous post described how to make an application HA and what exactly that means behind the scenes. This post is to augment the explanation in the HA PEP of how an administrator should expect to implement the routing layer for HA apps.

The routing layer implementation is currently left entirely up to the administrator. At some point OpenShift will likely ship a supported routing layer component, but the first priority was to provide an SPI (Service Provider Interface) so that administrators could reuse existing routing and load balancer infrastructure with OpenShift. Since most enterprises already have such infrastructure, we expected they would prefer to leverage that investment (both in equipment and experience) rather than be forced to use something OpenShift-specific.

Still, this leaves the administrator with the task of implementing the interface to the routing layer. Worldline published an nginx implementation, and we have some reference implementations in the works, but I thought I’d outline some of the details that might not be obvious in such an implementation.

The routing SPI

The first step in the journey is to understand the routing SPI events. The routing SPI itself is an interface on the OpenShift broker app that must be implemented via plugin. The example routing plugin that is packaged for Origin and Enterprise simply serializes the SPI events to YAML and puts them on an ActiveMQ message queue/topic. This is just one way to distribute the events, but it’s a pretty good way, at least in the abstract. For routing layer development and testing, you can just publishes messages on a topic on the ActiveMQ instance OpenShift already uses (for Enterprise, openshift.sh does this for you) and use the trivial “echo” listener to see exactly what comes through. For production, publish events to a queue (or several if multiple instances need updating) on an HA ActiveMQ deployment that stores messages to disk when shutting down (you really don’t want to lose routing events) – note that the ActiveMQ deployment described in OpenShift docs and deployed by the installer does not do this, being intended for much more ephemeral messages.

I’m not going to go into detail about the routing events. You’ll become plenty familiar if you implement this. You can see some good example events in this description, but always check what is actually coming out of the SPI as there may have been updates (generally additions) since. The general outline of the events can be seen in the Sample Routing Plug-in Notifications table from the Deployment Guide or in the example implementation of the SPI. Remember you can always write your own plugin to give you information in the desired format.

Consuming SPI events for app creation

The routing SPI publishes events for all apps, not just HA ones, and you might want to do something with other apps (e.g. implement blue/green deployments), but the main focus of a routing layer is to implement HA apps. So let’s look at how you do that. I’m assuming YAML entries from the sample activemq plugin below — if you use a different plugin, similar concepts should apply just with different details.

First when an app is created you’re going to get an app creation event:

$ rhc app create phpha php-5.4 -s

:action: :create_application
:app_name: phpha
:namespace: demo
:scalable: true
:ha: false

This is pretty much just a placeholder for the application name. Note that it is not marked as HA. There is some work coming to make apps HA at creation, but currently you just get a scaled app and have to make it HA after it’s created. This plugin doesn’t publish the app UUID, which is what I would probably do if I were writing a plugin now. Instead, you’ll identify the application in any future events by the combination of app_name and namespace.

Once an actual gear is deployed, you’ll get two (or more) :add_public_endpoint actions, one for haproxy’s load_balancer type and one for the cartridge web_framework type (and possibly other endpoints depending on cartridge).

:action: :add_public_endpoint
:app_name: phpha
:namespace: demo
:gear_id: 542b72abafec2de3aa000009
:public_port_name: haproxy-1.4
:public_port: 50847
- http
- ws
- load_balancer
- frontend: ''
 backend: ''
- frontend: /health
 backend: /configuration/health

You might expect that when you make the app HA, there is some kind of event specific to being made HA. There isn’t at this time. You just get another load_balancer endpoint creation event for the same app, and you can infer that it’s now HA. For simplicity of implementation, it’s probably just best to treat all scaled apps as if they were already HA and define routing configuration for them.

Decision point 1: The routing layer can either direct requests only to the load_balancer endpoints and let them forward traffic all to the other gears, or it can actually just send traffic directly to all web_framework endpoints. The recommendation is to send traffic to the load_balancer endpoints, for a few reasons:

  1. This allows haproxy to monitor traffic in order to auto-scale.
  2. It will mean less frequent changes to your routing configuration (important when changes mean restarts).
  3. It will mean fewer entries in your routing configuration, which could grow quite large and become a performance concern.

However, direct routing is viable, and allows an implementation of HA without actually going to the trouble of making apps HA. You would just have to set up a DNS entry for the app that points at the routing layer and use that. You’d also have to handle scaling events manually or from the routing layer somehow (or even customize the HAproxy autoscale algorithm to use stats from the routing layer).

Decision point 2: The expectation communicated in the PEP (and how this was intended to be implemented) is that requests will be directed to the external proxy port on the node (in the example above, that would be There is one problem with doing this – idling. Idler stats are gathered only on requests that go through the node frontend proxy, so if we direct all traffic to the port proxy, the haproxy gear(s) will eventually idle and the app will be unavailable even though it’s handling lots of traffic. (Fun fact: secondary gears are exempt from idling – doesn’t help, unless the routing layer proxies directly to them.) So, how do we prevent idling? Here are a few options:

  1. Don’t enable the idler on nodes where you expect to have HA apps. This assumes you can set aside nodes for (presumably production) HA apps that you never want to idle. Definitely the simplest option.
  2. Implement health checks that actually go to the node frontend such that HA apps will never idle. You’ll need the gear name, which is slightly tricky – the above endpoint being on the first gear, it will be accessible by a request for http://phpha-demo.cloud_domain/health to the node at When the next gear comes in, you’ll have to recognize that it’s not the head gear and send the health check to e.g. http://542b72abafec2de3aa000009-demo.cloud_domain/health.
  3. Flout the PEP and send actual traffic to the node frontend. This would be the best of all worlds since the idler would work as intended without any special tricks, but there are some caveats I’ll discuss later.

Terminating SSL (TLS)

When hosting multiple applications behind a proxy, it is basically necessary to terminate SSL at the proxy. (Despite SSL having been essentially replaced by TLS at this point, we’re probably going to call it SSL for the lifetime of the internet.) This has to do with the way routing works under HTTPS; during the intialization of the TLS connection, the client has to indicate the name it wants (in our case the application’s DNS name) in the SNI extension to the TLS “hello”. The proxy can’t behave as a dumb layer 4 proxy (just forwarding packets unexamined to another TLS endpoint) because it has to examine the stream at the protocol level to determine where to send it. Since the SNI information is (from my reading of the RFC) volunteered by the client at the start of the connection, it does seem like it would be possible for a proxy to examine the protocol and then act like a layer 4 proxy based on that examination, and indeed I think F5 LBs have this capability, but it does not seem to be a standard proxy/LB capability, and certainly not for existing open source implementations (nginx, haproxy, httpd – someone correct me if I’m missing something here), so to be inclusive we are left with proxies that operate at the layer 7 protocol layer, meaning they perform the TLS negotiation from the client’s perspective.

Edit 2014-10-08: layer 4 routing based on SNI is probably more available than I thought. I should have realized HAproxy 1.5 can do it, given OpenShift’s SNI proxy is using that capability. It’s hard to find details on though. If most candidate routing layer technologies have this ability, then it could simplify a number of the issues around TLS because terminating TLS could be deferred to the node.

If that was all greek to you, the important point to extract is that a reverse proxy has to have all the information to handle TLS connections, meaning the appropriate key and certificate for any requested application name. This is the same information used at the node frontend proxy; indeed, the routing layer will likely need to reuse the same *.cloud_domain wildcard certificate and key that is shared on all OpenShift nodes, and it needs to be made aware of aliases and their custom certificates so that it can properly terminate requests for them. (If OpenShift supported specifying x509 authentication via client certificates [which BTW could be implemented without large structural changes], the necessary parameters would also need to be published to the routing layer in addition to the node frontend proxy.)

We assume that a wildcard certificate covers the standard HA DNS name created for HA apps (e.g. in this case ha-phpha-demo.cloud_domain, depending of course on configuration; notice that no event announces this name — it is implied when an app is HA). That leaves aliases which have their own custom certs needing to be understood at the routing layer:

$ rhc alias add foo.example.com -a phpha
:action: :add_alias
:app_name: phpha
:namespace: demo
:alias: foo.example.com

$ rhc alias update-cert foo.example.com -a phpha --certificate certfile --private-key keyfile
:action: :add_ssl
:app_name: phpha
:namespace: demo
:alias: foo.example.com
:ssl: [...]
:private_key: [...]

Aliases will of course need their own routing configuration entries regardless of HTTP/S, and something will have to create their DNS entries as CNAMEs to the ha- application DNS record.

A security-minded administrator would likely desire to encrypt connections from the routing layer back to the gears. Two methods of doing this present themselves:

  1. Send an HTTPS request back to the gear’s port proxy. This won’t work with any of the existing cartridges OpenShift provides (including the haproxy-1.4 LB cartridge), because none of them expose an HTTPS-aware endpoint. It may be possible to change this, but it would be a great deal of work and is not likely to happen in the lifetime of the current architecture.
  2. Send an HTTPS request back to the node frontend proxy, which does handle HTTPS. This actually works fine, if the app is being accessed via an alias – more about this caveat later.

Sending the right HTTP headers

It is critically important in any reverse-proxy situation to preserve the client’s HTTP request headers indicating the URL at which it is accessing an application. This allows the application to build self-referencing URLs accurately. This can be a little complicated in a reverse-proxy situation, because the same HTTP headers may be used to route requests to the right application. Let’s think a little bit about how this needs to work. Here’s an example HTTP request:

POST /app/login.php HTTP/1.1
Host: phpha-demo.openshift.example.com

If this request comes into the node frontend proxy, it looks at the Host header, and assuming that it’s a known application, forwards the request to the correct gear on that node. It’s also possible (although OpenShift doesn’t do this, but a routing layer might) to use the path (/app/login.php here) to route to different apps, e.g. requests for /app1/ might go to a different place than /app2/.

Now, when an application responds, it will often create response headers (e.g. a redirect with a Location: header) as well as content based on the request headers that are intended to link to itself relative to what the client requested. The client could be accessing the application by a number of paths – for instance, our HA app above should be reachable either as phpha-demo.openshift.example.com or as ha-phpha-demo.openshift.example.com (default HA config). We would not want a client that requests the ha- address to receive a link to the non-ha- address, which may not even resolve for it, and in any case would not be HA. The application, in order to be flexible, should not make a priori assumptions about how it will be addressed, so every application framework of any note provides methods for creating redirects and content links based on the request headers. Thus, as stated above, it’s critical for these headers to come in with an accurate representation of what the client requested, meaning:

  1. The same path (URI) the client requested
  2. The same host the client requested
  3. The same protocol the client requested

(The last is implemented via the “X-Forwarded-Proto: https” header for secure connections. Interestingly, a recent RFC specifies a new header for communicating items 2 and 3, but not 1. This will be a useful alternative as it becomes adopted by proxies and web frameworks.)

Most reverse proxy software should be well aware of this requirement and provide options such that when the request is proxied, the headers are preserved (for example, the ProxyPreserveHost directive in httpd). This works perfectly with the HA routing layer scheme proposed in the PEP, where the proxied request goes directly to an application gear. The haproxy cartridge does not need to route based on Host: header (although it does route requests based on a cookie it sets for sticky sessions), so the request can come in for any name at all and it’s simply forwarded as-is for the application to use.

The complication arises in situations where, for example, you would like the routing layer to forward requests to the node frontend proxy (in order to use HTTPS, or to prevent idling). The node frontend does care about the Host header because it’s used for routing, so the requested host name has to be one that the OpenShift node knows in relation to the desired gear. It might be tempting to think that you can just rewrite the request to use the gear’s “normal” name (e.g. phpha-demo.cloud_domain) but this would be a mistake because the application would respond with headers and links based on this name. Reverse proxies often offer options for rewriting the headers and even contents of responses in an attempt to fix this, but they cannot do so accurately for all situations (example: links embedded in JavaScript properties) so this should not be attempted. (Side note: the same prohibition applies to rewriting the URI path while proxying. Rewriting example.com/app/… to app.internal.example.com/… is only safe for sites that provide static content and all-relative links.)

What was that caveat?

I mentioned a caveat both on defeating the idler and proxying HTTPS connections to the node frontend, and it’s related to the section above. You can absolutely forward an HA request to the node frontend if the request is for a configured alias of the application, because the node frontend knows how to route aliases (so you don’t have to rewrite the Host: header which, as just discussed, is a terrible idea). The caveat is that, strangely, OpenShift does not create an alias for the ha- DNS entry automatically assigned to an HA app, so manual definition of an alias is currently required per-app for implementation of this scheme. I have created a feature request to instate the ha- DNS entry as an alias, and being hopefully easy to implement, this may soon remove the caveat behind this approach to routing layer implementation.

Things go away too

I probably shouldn’t even have to mention this, but: apps, endpoints, aliases, and certificates can all go away, too. Make sure that you process these events and don’t leave any debris lying around in your routing layer confs. Gears can also be moved from one host to another, which is an easy use case to forget about.

And finally, speaking of going away, the example routing plugin initially provided :add_gear and :remove_gear events, and for backwards compatibility it still does (duplicating the endpoint events). These events are deprecated and should disappear soon.

HA applications on OpenShift

Beginning with version 2.0, OpenShift Enterprise supports making applications “highly available”. However, there is very little documentation on the details of implementing HA apps or exactly how they work. We surely have customers using these, but they’re mostly doing so with a lot of direct help from our consulting services. Nothing in the documentation brings it all together, so I thought I’d share what I know at this point. I think we can expect these implementation details to remain fairly stable for the lifetime of OSE 2.

This is kind of a brain dump. We are working to get most of this into the manuals if it isn’t already (at least what’s fit for it and not just me rambling). But I don’t know anywhere else to find all this in one place right now.

Why and how?

The motivation behind supplying this feature and the basic architectural points and terminology are well covered at the start of the HA PEP. Definitely read this for background if you haven’t already; I won’t repeat most of the points.

As a super quick overview: HA is an enhancement to scaled apps. Standard scaled apps allow easy horizontal scaling by deploying duplicate instances of the application code and framework cartridge in multiple gears of your app, accessed via a single load balancer (LB) proxy gear (LB is performed by the haproxy cartridge in OSE 2). For a scaled app to become HA, it should have two or more LB gears on different nodes that can each proxy to the gears of the app. That way, if a node containing one LB gear goes down, you can still reach the app via another. This only makes the web framework HA — shared storage and DB replication aren’t OpenShift features (yet), so practically speaking. your app should be stateless or use external HA storage/DB if you actually want it to be HA.

So how do we direct HTTP requests to the multiple LB gears? Via the routing layer, which is described in the PEP, but which OpenShift does not (yet) supply a component/configuration for. If you are interested in HA apps, then chances are good you already have some kind of solution deployed for highly-available routing to multiple instances of an application (e.g. F5 boxes). All you need to do is hook OpenShift and your existing solution together via the Routing System Programming Interface (SPI). Once this is configured, HA apps are given an alias in DNS that resolves to the routing layer, which proxies requests to the LB gears on OpenShift nodes, which proxy requests to the application gears.

Making an app HA (the missing User Guide entry)

The OSE admin guide indicates how administrators can configure OSE to allow HA apps and enable specific users to make apps HA. I’ll assume you have a deployment and user with this capability.

To make an HA app, you first create a scaled app, then make it HA. The “make-ha” REST API call is implemented as an event (similar to “start” or “stop”) on an application. Direct REST API access is currently the only way to do this – rhc and other tools do not implement this call yet. So, currently the only mention is in the REST API guide. So, for example:

$ rhc create-app testha ruby-1.9 -s

Your application ‘testha’ is now available.

$ curl -k -X POST https://broker.example.com/broker/rest/domains/test/applications/testha/events –user demo:password –data-urlencode event=make-ha

long JSON response including: “messages”:[{“exit_code”:0, “field”:null, “index”:null, “severity”:”info”, “text”:”Application testha is now ha”}], “status”:”ok”

Upon success, the app will have scaled to two gears, and your second gear will also be a LB gear. You can confirm by sshing in to the second gear and looking around.

$ rhc app show –gears -a testha

ID State Cartridges Size SSH URL
53c6c6b2e659c57659000002 started haproxy-1.4 ruby-1.9 small 53c6c6b2e659c57659000002@testha-test.example.com
53c6c74fe659c57659000021 started haproxy-1.4 ruby-1.9 small 53c6c74fe659c57659000021@53c6c74fe659c57659000021-test.example.com

$ ssh 53c6c74fe659c57659000021@53c6c74fe659c57659000021-test.example.com

> ls

app-deployments app-root gear-registry git haproxy ruby

> ps -e

12607 ? 00:00:00 haproxy
13829 ? 00:00:00 httpd


A third gear will not look the same; it will only have the framework cartridge. I should mention that at this moment there’s a bug in rhc such that it displays all framework gears as having the haproxy LB cartridge, when actually only the first two do.

What does make-ha actually do?

Behind the scenes are a few critical changes from the make-ha event.

First, a new DNS entry has been created to resolve requests to the router. How exactly this happens depends on configuration. (Refer to the Admin Guide for details on how the HA DNS entry is configured.) In the simplest path (with MANAGE_HA_DNS=true), OpenShift itself creates the DNS entry directly; the default is just to prepend  “ha-” to the app name and point that entry at ROUTER_HOSTNAME. Thus our app above would now have a DNS entry ha-testha-test.example.com. With MANAGE_HA_DNS=false, OpenShift counts on the routing layer to receive the routing SPI event and create this DNS entry itself.

In either case, this DNS entry is only useful if it points at the router which serves as a proxy. A request to one of the nodes for “ha-testha-test.example.com” would not be proxied correctly – it’s supposed to be relayed by the routing layer as a request for either “testha-test.example.com” or a secondary LB gear. It’s also possible for the router to just proxy directly to the individual gears (endpoints for which are provided in the routing SPI); however, this may not be desirable for a variety of reasons.

A second change that occurs is that the parameters for the haproxy cartridge are modified. By default in a scaled application, the haproxy cartridge manifest has a minimum and maximum scale of 1 (see the manifest), so there will always be exactly one. But when you make an app HA, in the MongoDB record for the haproxy cartridge instance in your app, the minimum is changed to 2, and the maximum is changed to -1 (signifying no maximum). Also the HA multiplier is applied, which I’ll discuss later. As a result of raising the minimum to 2, the app scales up one gear to meet the minimum.

There are some interesting oddities here.

First, I should note that you can’t scale the haproxy cartridge directly via the REST API at all. You’ll just get an error. You only have the ability to scale the framework cartridge, and the haproxy cartridge may be deployed with it.

Also, once your app is HA, you can no longer scale below two gears:

$ rhc scale-cartridge -a testha ruby –min 1 –max 1
Setting scale range for ruby-1.9 … Cannot set the max gear limit to ‘1’ if the application is HA (highly available)

By the way, there is no event for making your app not HA. Just destroy and re-create it.

Also, if you already scaled the framework cartridge to multiple gears, then making it HA will neither scale it up another gear, nor deploy the haproxy cartridge on an existing gear (which is what I would have expected). So it will not actually be HA at that point. Instead, an haproxy cartridge will be deployed with the next gear created. If you then scale down, that gear will be destroyed, and your app again effectively ceases to be HA (Edit: this is considered a bug). So, make sure you make your app HA before scaling it, so you are assured of having at least two LB gears. A little more about this at the end.

How does auto-scaling work with HA?

If you’re familiar with OpenShift’s auto-scaling, you may be wondering how two or more LB gears coordinate traffic statistics in order to decide when to scale.

First, I’ll say that if you’re going to the trouble to front your OpenShift app with an expensive HA solution, you may want to disable auto-scaling and let your router track the load and decide when to scale. Practically speaking, that can be tricky to implement (there actually isn’t an administrative mechanism to scale a user-owned app, so workarounds are needed, such as having all HA apps owned by an “administrative” user), but it’s something to think about.

That said, if you’re using auto-scaling, perhaps with a customized algorithm, you’ll want to know that the standard haproxy scaling algorithm runs in a daemon on the first gear (“head” gear) of the app, in both regular scaled apps and HA apps. In either case, it bases scaling decisions on a moving average of the number of HTTP sessions in flight. The only difference with an HA app is that the daemon makes a request each sampling period to all of the other LB gears to gather the same statistic and add it into the average.

Also, auto-scaling does not occur if the head gear is lost. So, auto-scaling is not HA – another reason to consider manual scaling. Which brings me to…

What happens when a gear in an HA app is lost?

If a node crashes or for some other reason one or more gears of your HA app is out of commission (rack PDU cut out? Someone tripped over the network cable?), how does your app react? Well, this is no different from a regular scaled app, but it bears repeating: nothing happens. Remaining LB gears notice the missing gears are failing health checks and pull them out of rotation – and that’s it. There is no attempt to replace the missing gear(s) even if they are LB gears, or to augment the gear count to make up for them. No alerts are sent. As far as I’m aware, nothing even notifies the broker to indicate that a gear’s state has changed from “started” to “unresponsive” so it’s hard for the app owner to even tell.

The whole point of HA apps is to remove the single point of failure in standard scaled apps by providing more than one endpoint for requests, instead of the app being unreachable when the head LB gear is lost. However, there are still some functions performed only by the head gear:

  1. Auto-scaling
  2. Deployment of application code changes

These functions are still not HA in an HA app. They still rely on the head gear, and there’s no mechanism for another gear to take over the head gear designation. What’s more, if a gear is out of commission when an application update is deployed, there isn’t any mechanism currently for automatically updating the gear when it returns (the next deploy will bring it in sync of course). These are all known problems that are in our backlog of issues to address, but aren’t terribly obvious to those not intimately familiar with the architecture.

The general assumption in OpenShift when a node disappears is that it will probably come back later – either after some crisis has passed, or via resurrection from backups. So, there is no attempt at “healing” the app. If, however, a node goes away with no hope for recovery, there is a tool oo-admin-repair which can do a limited amount of cleanup to make the MongoDB records accurately reflect the fact that the gears on that node no longer exist. I’m not sure the extent to which scaled apps are repaired; I would expect that the list of gears to route to is accurately updated, but I would not expect the head gear to be replaced (or re-designated) if lost, nor any gears that aren’t scaled (databases or other services that don’t scale yet). If the head gear is intact and auto-scaling in use, then scaling should continue normally. If auto-scaling is disabled, I would not expect any scaling to occur (to replace missing gears) without the owner triggering an event. I haven’t rigorously researched these expectations, however. But let’s just say OpenShift nodes aren’t ready to be treated like cattle yet.

How do I know my gears are spread out enough?

Under OSE 2.0, gears gained anti-affinity, meaning that gears of a scaled app will avoid being placed together on the same node if there are other nodes with available capacity. OSE 2.1 provided more explicit mechanisms for controlling gear placement. The most relevant one here is putting nodes in availability zones — for example, you could use a different zone for each rack in your datacenter. If possible, OpenShift will place the gears of an HA app across multiple zones, ensuring availability if one entire zone of nodes is lost at once.

To have better assurance than “if possible” on your HA app spanning zones, take a look at the two settings in the broker mcollective plugin that force your app to do so.

There’s one major caveat here. Anti-affinity only applies when a gear is placed. The OpenShift administrators could move gears between nodes after the fact, and if they’re not being careful about it, they could move the LB gears of an HA app all to the same node, in which case it would effectively cease to be HA since one node outage would leave the app unavailable. There’s really nothing to prevent or even detect this circumstance at this time.

Verifying that your gears actually are spread across zones as expected is currently a laborious manual operation (list all the gears, see which node their addresses resolve to, check which zones the nodes are in), but the REST API and the tools that access it are being expanded to expose the zone and region information to make this easier.

Let me add one more bit of direction here regarding zones, regions, and districts. Although you need to create districts, and although you use the district tool to assign regions and zones to your nodes, there really is no relationship between districts and zones/regions. They exist for (almost) orthogonal purposes. The purpose of districts is to make moving gears between nodes easy. The purpose of zones and regions is to control where application gears land with affinity and anti-affinity. Districts can contain nodes of multiple zones, zones can span multiple districts. It actually makes a lot of sense to have multiple zones in a district and zones spanning districts, because that way if in the future you need to shut down an entire zone, it will be easy to move all of the gears in the zone elsewhere, because there will be other zones in the same district. OTOH, while you *can* have multiple regions in a district, it doesn’t make a lot of sense to do so, because you won’t typically want to move apps en masse between regions.

The scaling multiplier: more than 2 LB gears

Of course, with only two LB gears, it only takes two gear failures to make your app unreachable. For further redundancy as well as extra load balancing capacity, you might want more than the two standard LB gears in an HA app. This can be accomplished with the use of a (currently esoteric) setting called the HA multiplier. This value indicates how many of your framework gears should also be LB gears. A multiplier of 3 would indicate that every 3rd gear should be a LB gear (so, after 6 gears were created with the first two being LBs, the 9th, 12th, and so on would all be LBs). A multiplier of 10 would mean gears 30, 40, 50,… would be LB gears.

The default multiplier is 0, which indicates that after the first two LB are created to satisfy the minimum requirement for the HA app (remember, the haproxy minimum scale is changed to 2 for an HA app), no more will ever be created. The multiplier can be configured in two ways:

  1. Setting DEFAULT_HA_MULTIPLIER in broker.conf – currently undocumented, but it will be read and used if present. This sets the default, of course, so it is only relevant at the time an app is created. Changing it later doesn’t affect existing apps.
  2. Using oo-admin-ctl-app to change the multiplier for a specific app, e.g:

# oo-admin-ctl-app -l demo -a testha -c set-multiplier –cartridge haproxy-1.4 –multiplier 3

Note that both are administrative functions, not available to the user via the REST API.

The gear-to-LB ratio: rotating out the LB gears

At app creation, LB gears also directly (locally) serve application requests via the co-located framework cartridge. Once a scaled app is busy enough, gears serving as load balancers may consume significant resources just for the load balancing, leaving less capacity to serve requests. Thus, for performance reasons, after a certain limit of gears created in an app, LB gears remove themselves from the load balancing rotation (“rotate out”). Under this condition, the framework cartridge is left running on LB gears, but receives no requests.

The limit at which this occurs is governed by an environment variable OPENSHIFT_HAPROXY_GEAR_RATIO which is read by the haproxy cartridge. The idea is that when the ratio of total gears to LB gears (rounded) reaches this limit, that is when the LB gears rotate themselves out.

The default value is 3. So in a plain scaled app, once the third gear is created, the first gear is removed from rotation. In an HA app, once the 5th gear is created (5 gears / 2 LB gears rounds to 3) the first two gears are removed from rotation (resulting in an actual reduction from 4 gears servicing requests to 3). In general, it would be unwise to set this value higher than the HA multiplier, as the LB gears would be rotated in and out unevenly as the app scaled.

The most obvious way to change this value is to set it node-wide in /etc/openshift/env/OPENSHIFT_HAPROXY_GEAR_RATIO (an administrative action). However, being an environment variable, it’s also possible for the user to override it by setting an environment variable for an individual app (there’s no administrative equivalent).


The primary method of integrating a routing layer into OpenShift is via the routing SPI, a service that announces events relevant to a routing layer, such as gears being created. The Routing SPI is an interface in the broker application that may be implemented via a plugin of your choosing. The documentation describes a plugin implementation that relays the events to an ActiveMQ messaging broker, and then an example listener for consuming those messages from the ActiveMQ broker. It is important to note that this is only one implementation, and can be substituted with anything that makes sense in your environment. Publishing to a messaging bus is a good idea, but note that as typically deployed for OpenShift, ActiveMQ is not configured to store messages across restarts. So, definitely treat this only as informational, not a recommended implementation.

Another method for integrating a routing layer is via the REST API. Because the REST API has no administrative concept and access is restricted per user, the REST API is somewhat limited in an administrative capacity; for example, there is no way to list all applications belonging to all users. However querying the REST API could be suitable when implementing an administrative interface to OpenShift that intermediates all requests to the broker, at least for apps that require the routing layer (i.e. HA apps). For example, there could be a single “administrative” user that owns all production applications that need to be HA, and some kind of system for regular users to request apps via this user. In such cases, it may make sense to retrieve application details synchronously while provisioning via the REST API, rather than asynchronously via the Routing SPI.

The most relevant REST API request here is for application endpoints. This includes the connection information for each exposed cartridge on each gear of a scaled app. For example:

$ curl -k -u demo:password https://broker.example.com/broker/rest/domains/test/applications/testha/gear_groups?include=endpoints | python -mjson.tool

... ( example LB endpoint: )
   "cartridge_name": "haproxy-1.4", 
   "external_address": "", 
   "external_port": "56432", 
   "internal_address": "", 
   "internal_port": "8080", 
   "protocols": [
   "types": [


Odds and ends

It’s perhaps worth talking about the “gear groups” shown in the API response a little bit, as they’re not terribly intuitive. Gear groups are sets of gears in the app that replicate the same cartridge(s). For a scaled app, the framework cartridge would be in one gear group along with the LB cartridge that scales it. The haproxy LB cartridge is known as a “sparse” cartridge because it is not deployed in all of the gears in the group, just some. If a database cartridge is added to the app, that is placed in a separate gear group (OpenShift doesn’t have scaling/replicated DB cartridges yet, but when it does, these gears groups will become larger than one and will “scale” separately from the framework cartridge). Without parameters, the gear_groups REST API response doesn’t indicate which cartridge is on which gear, just the cartridges that are in a gear group and the gears in that group; this is why rhc currently indicates the haproxy is located on all framework gears. This will be fixed by specifying the inclusion of endpoints in the request.

Remember how making an app HA makes it scale up to two LB gears, but not it if it’s already two gears? I suspect the “minimum” scale from the cartridge is applied to the gear group, without regard for the actual number of cartridge instances; so, while making an app HA will set the haproxy cartridge “minimum” to 2, the effect is that the gear group scale minimum is set to 2 (i.e. scale is a property of the gear group). If the gear group has already been scaled to 2 before being made HA, then it doesn’t need to scale up to meet the minimum (and in fact, won’t allow scaling down, so you can’t ever get that second gear to have the haproxy LB).

One thing discussed in the PEP that has no implementation (aside from a routing layer component): there’s no mechanism for a single LB gear to proxy only to a subset of the other gears. If your app is scaled to 1000 gears, then each LB gear proxies to all 1000 gears (minus LB gears which we assume are rotated out). A few other pieces of the PEP are not complete as yet: removing or restarting specific gears, removing HA, and probably other bits planned but not delivered in the current implementation.


The 7 easy steps TIME VAMPIRES hope you won’t use. #6 changed my life!

Vampire Kitty

Picture by Faris Algosaibi, under CC by 2.0 license (https://creativecommons.org/licenses/by/2.0/). Cropped from original.

You sit down after dinner to research a few home improvement ideas on the web. A funny picture with a link catches your eye and… when bed time rolls around you’ve read fourteen motivational stories, watched a series of skateboarding turkey videos, grumbled on Facebook about obviously corrupt politicians in another state, shared some really pretty pictures of Antarctica you found, and not done any research! How did this happen? How does this keep happening?

How well you adapt for the rest of the 21st century will depend mainly on how well you defend against the distractions and outright manipulation in the increasing stream of available information. With the rise of social media, the voices screaming “look at me!” have shifted tactics – not only are they trying to get your attention, they’re trying to hijack your friends list. It didn’t take long to discover how to push our buttons; the psychological techniques are becoming well-known and ubiquitous. Every social media whore dreams of “going viral” and will post any outrageous thing if it gets a few million hits to their crappy ad farm.

“Go viral” is a good term for it; these are attention viruses spread via social media, using your resources as a host body to spread themselves. In addition to wasting your time, they tend to be inflammatory, misleading, and low in information. They make the internet a worse place. And I, for one, have gotten sick of seeing the same kinds of headlines trying to suck me in, the same kind of misdirection and manipulation stirring the internet’s collective psyche.

The good news is that you can fight back. The enemy may have your number, but you can learn to recognize the manipulation and avoid it. Here are some current ways – but rest assured that their bag of tricks will continue to adapt (you might want to run a search every so often for the latest).

1. Avoid posts with numbers in the title

Perhaps it’s the sense of accomplishment we feel as we tick through the list. Perhaps it’s the curiosity to see “are there *really* (only) that many?” Perhaps it’s just to see if we’re already smart enough to know the whole list. Maybe numbers are inherently more credible. Whatever the reason, social media experts have figured out that we will pretty much click on (and share) any damn thing presented as a numbered list. Even if you know it will probably tell you nothing new. (Numbered lists aren’t the only offenders; consider, for example, the trend of “99% will fail this simple test!” articles.)

Actual news is not presented as “25 things you didn’t know about XYZ”. Useful advice is not presented as “4 easy steps”. In fact, it’s incredibly rare for it to have any number at all in the title. So if it does, that’s a red flag: you can feel simultaneously smug and smarter about skipping this link because it is undoubtedly an attention virus.

Especially with an addendum like “Number X blew my mind!”

2. Avoid titles with emotionally charged words

“Shocking!” “Amazing!” “You won’t believe…” “Blown away” “Can’t stop laughing/crying” “Epic” ALL CAPS!

These and other attention-seeking techniques are used exclusively by attention-desperate social media whores. Not by actually informative articles. Not by people who have a purpose for writing other than maximizing clicks and shares. It is manipulation, pure and simple. The more prevalent this sort of thing becomes, the more it drowns out balanced, informative writing on the internet. And the more you read it, the more often your blood pressure and cortisol levels will rise needlessly. Don’t click that link! Don’t do it!

3. Avoid posts that you react to with “no way!” or “oh yeah?”

If you can feel your eyebrows rising just from reading the headline (police brutality, stupid politician tricks, “you’ll never guess”), you can bet it’s deliberately misleading in order to shock you and draw you in. Resist. And for the love of Bog, don’t get drawn into threads 100 comments long by people who didn’t even read the article. You will accomplish nothing but making yourself and perhaps a few others angry. Don’t bother unless you’re a troll and that’s how you get your lulz.

4. If you find yourself reading an attention virus, at least avoid sharing it

So you might enjoy following Facebook links for some meaningless entertainment from time to time. I get it. But… do you have to infect your friends too? Do you have to reward these time vampires?

No. You don’t. In fact, with the Information Deluge unfolding this century, it’s your responsibility not to.

If (perhaps by accident) you find yourself visiting one of these, at least keep it to yourself. You cover your mouth when you cough and sneeze, right? Have the same courtesy for your readers’ brains. Or are you one of those people still forwarding stupid chain letters?

5. Use ad-blocking browser plugins

The push for sensationalizing the internet is all about displaying ads. More clicks mean more ad views and more revenue. If you kill the ad revenue, you stop rewarding the behavior.

Also, you don’t need to waste your attention on ads. I have basically never seen an ad on Facebook. I don’t really see any ads on the web other than the occasional text ad or YouTube intro. How? I’ve been using ad-blocking software since it was an esoteric art requiring running a local proxy and manually tweaking browser settings and blacklist entries.

It’s a lot easier now. For years it’s been as easy as searching for browser plugins and clicking a few buttons to install them. I know about AdBlock+ for FireFox and AdBlock+ for Chrome. If you’re using something else there’s probably a plugin for that too (even for mobile).

Personally, I think advertising funding is a blight on the internet (“you won’t believe” I once worked for an advertising startup) and would like to see it destroyed in favor of other models. If you disagree and feel morally obliged to allow websites to manipulate your mind in return for the benefit they bring you (or even find targeted ads actually – choke! – informative), you can usually configure an ad-blocker to allow ads selectively, and still starve the time vampires you stumble upon.

6. Use site-blocking browser plugins

Self-restraint is a great thing to cultivate, but most of us would admit to needing a little help. And the wonderful thing about the information age is that there are tools to help you… automatically.

You can use site-blocker plugins to block whole sites that you know are full of trash, or just to restrict the amount of time that you waste at them. For example, BlockSite for Chrome and BlockSite for Firefox can block sites at specified times and days of the week. Also consider StayFocusd for Chrome, which tracks and limits the time you waste as well as providing a “nuclear option” to block all “bad” sites or allow only “good” sites for a few hours to help you concentrate. LeechBlock for Firefox appears similar. These double as procrastination-blockers, useful beyond simple avoidance of attention viruses.

Consider blocking all of the most addictive sites on the internet or pretty much anything linked from Buzzfeed (they recommend blocking themselves!). Or just look through your browser history to see where the time goes.

7. Filter your news sources

The easiest way to save money is to have it automatically deducted from your paycheck. You don’t miss the money you never see. Similarly, the easiest way to reserve your attention for worthy topics is to block ones you know tend to be trash. You don’t have to decide to ignore the tripe you never see.

Spam, trolls, and general “noise” have all been with us since the dawn of the internet. News-readers on usenet used killfiles to automatically ignore posts. Once email became a cesspool of spam and phishing, filtering became a standard tool there too (some services automating it with great sophistication). Social networking may take a little longer because frankly, Facebook’s profits are built on sucking you in and using your friends list and interests to advertise to you. It’s unlikely they’ll be providing useful automated filters of any variety soon.

Sick of the clutter on Facebook? Try F.B. Purity. It’s a browser plugin that essentially rewrites the Facebook interface, allowing you to filter out what you don’t want (hopefully this post has given you some good ideas). It’s pretty easy to install; just be aware that Facebook’s interface is changing all the time, and when it does you may experience bizarre glitches due to a mismatch between what Facebook provides and what F.B. Purity expects, at which point you’ll need to (1) recognize what’s going on (2) possibly wait for an update from FBP or just disable it for a while, and (3) update FBP. So this isn’t for everyone, but it’s what’s available right now. Perhaps other social networks that aren’t as invested in cramming junk in your face will lead the way in enabling filtering, forcing Facebook to do the same. Or perhaps Facebook will become irrelevant. I don’t know what will happen, but if users don’t start valuing the ability, it won’t appear of its own accord. I suggest taking matters into your own hands.

Other sources often do provide methods of filtering, or there may be a browser plugin to enable it. Search for these and use them.


Am I aware of the irony/hypocrisy inherent in this post at multiple levels? Yes; yes, I am.

But now you know. If you make this post the last time you’re ever roped in by these tactics, I can die happy.

Now share this with all your friends, and leave some comments below!


Get every new post delivered to your Inbox.

Join 205 other followers