Working around the mysterious yum multilib error on an OpenShift Enterprise install

This one took a little while to track down and understand so I offer it here in the spirit of helping someone else get around this without that discovery process.

TL;DR

if when installing cartridges on an OpenShift Enterprise node host you get a yum error with “Error:  Multilib version problems found.” then make sure you have followed the steps in https://access.redhat.com/site/articles/316613 even if you don’t want the JBoss cartridges.

The problem

The example configure script we’ve provided for OpenShift Enterprise currently installs node cartridges as a single yum command which attempts to install all of them, but hedges its bets with the –skip-broken flag in case something doesn’t work out with one or more cartridges. This keeps you from having an install where no cartridges get installed just because some fiddling dependency problem blocked one of them and the others could have proceeded fine. This is very helpful during our development testing. For a production deployment it may or may not be, depending on how you define “helpful.”

So, I found that if I ran a production configuration and did not follow the directions at https://access.redhat.com/site/articles/316613 to work around the nature of our JBoss subscription dependencies, not only did JBoss cartridge installation fail, but I got this bizarre yum error message:

Error: Multilib version problems found. This often means that the root
cause is something else and multilib version checking is just
pointing out that there is a problem.

[…]

Protected multilib versions: zlib-1.2.3-27.el6.i686 != zlib-1.2.3-29.el6.x86_64

(The full message is similar to this post.) Reading between the lines there, what this error tells you is: There is some kind of dependency problem, we found it when trying to update zlib (and found that we were trying to install mismatched versions on the two architectures i686 and x86_64), but the real problem is probably somewhere way upstream. Which is about as useless an error message as you can get, but I understand where these things come from – it’s pretty hard to unravel the chain of consequences sometimes to give a helpful error message.

So, what the heck, right?

Also, if you try to install individual cartridges, they’ll work fine. If you remove the JBoss ones from the yum command, it works fine. It only fails on exactly the command that is run by default by the script.

Debugging

Running yum with the -v flag (verbose) gives a lot more info. I did not chase this down to the exact dependency chain but I gather the following describes what’s happening.

The first annoying question is, why is it even trying to install the i686 zlib in the first place? Everything I’m using is 64-bit. The next question is, why is it trying to install different versions? The -27 release is available for both architectures, why isn’t it just using that?

The clue comes in these lines of debugging output:

SKIPBROKEN: removing zlib-devel-1.2.3-29.el6.x86_64 from transaction 
SKIPBROKEN: removing zlib-devel-1.2.3-29.el6.x86_64 from pkgSack & updates

zlib-devel requires a zlib with the same version, and itself is required by freetype-devel, a requirement of the python cartridge.  It is also a dependency for libxml2-devel (required from the ruby cartridge) and openssl-devel (distant requirement for several cartridges). So let’s just say it is in the thick of things.

Unraveling the chain

I gather that what happened is this: With the channels broken, yum got deep into resolution and realized it could not install the JBoss cartridges. With the –skip-broken flag, yum removed from the transaction those cartridges and all of the package versions it had included for their dependencies; then it tried to resolve dependencies for the rest of the transaction with ALL of those packages not allowed in the transaction (as any could be the source of the problem). Apparently zlib-devel-1.2.3-29.el6.x86_64 was part of what was excised. But zlib-devel was still needed by other cartridges, and this is where it gets really bizarre.

openssl-devel-1.0.0-27.el6_4.2.x86_64 requires: zlib-devel
–> Processing Dependency: zlib-devel for package: openssl-devel-1.0.0-27.el6_4.2.x86_64
Searching pkgSack for dep: zlib-devel
TSINFO: Marking zlib-devel-1.2.3-27.el6.x86_64 as install for openssl-devel-1.0.0-27.el6_4.2.x86_64
[…]
—> Package zlib-devel.x86_64 0:1.2.3-27.el6 will be installed

[…]

zlib-devel-1.2.3-27.el6.x86_64 requires: zlib = 1.2.3-27.el6
–> Processing Dependency: zlib = 1.2.3-27.el6 for package: zlib-devel-1.2.3-27.el6.x86_64

Searching pkgSack for dep: zlib
Potential resolving package zlib-1.2.3-27.el6.x86_64 has newer instance installed.
TSINFO: Marking zlib-1.2.3-27.el6.i686 as install for zlib-devel-1.2.3-27.el6.x86_64
–> Running transaction check
—> Package zlib.i686 0:1.2.3-27.el6 will be installed

Because zlib-devel -29 was verboten, it fell back to zlib-devel -27. But that requires zlib -27, and -29 was already installed. You can’t have both installed at the same time, but yum saw a way to resolve the request by pulling in the zlib package from the i686 arch (zlib-devel did not specify arch in its requirement). So now yum wanted to install zlib-1.2.3-27.i686 on a system where zlib-1.2.3-29.x86_64 was already installed. It wasn’t allowed to downgrade the existing installation and can’t find any other way to satisfy the requirements under the restrictions imposed by having removed part of the transaction. So finally yum gave up and I got this confusing error about the multilib versions not matching, far from the source of the problem (which is, arguably, yum’s process for dependency resolution following pruning by –skip-broken).

Fixing the repo configuration (https://access.redhat.com/site/articles/316613) so that JBoss cartridges install cleanly fixes the problem. Not attempting to install them also fixes the problem.

Hopefully this provides some useful insight into this kind of problem, even for those who arrive here by completely different means.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: