XenServer, Patch 22 and Crypto

The What

I am using XenServer as my private solution for my network. It’s fast, reliable, open-source and free (as in free beer). I am sort of a fanboy. That said we are using XenServer at work, too.

Somewhat recently Citrix, maker of XenServer released hotfix XS65ESP1022 aka Patch 22, release notes:

This hotfix supports the upgrade of OpenSSL package to version 1.0.1.

Files Updated

stunnel-4.15-17.x86_64.rpm
make-3.81-3.el5.x86_64.rpm
openssl-wrapper-0.1-59.x86_64.rpm
openssl-xs-1.0.1e-42.xs15.x86_64.rpm
ca-certificates-2012.87-1.noarch.rpm
openssl-xs-libs-1.0.1e-42.xs15.x86_64.rpm
openvswitch-2.1.3-13.7579.x86_64.rpm
xenserver-transfer-vm-6.5.0-116122c.noarch.rpm

The bold one however, introduces some issues. If you (like everyone) installed extra packages in the Dom0 (the hypervisor) and maybe even used packages from epel then stuff will break apart. For example:

  • Puppet (from epel, not puppetlabs repo) will be unable to create new certificates or segfault during a run,
  • Bacula and Bareos will launch but crash on Data transfer. This is evil because a “status client” command always succeeds.

This is because the wrapper now points to newer OpenSSL libraries from the XenServer Patch; the binaries are however compiled against a lesser version wrecking all kinds of havoc. And since Citrix has the stance of saying “Don’t install stuff inside the Dom0” and Epel wont recompile against the libraries from Citrix this is now a stalemate without an escape. Let’s build one.

The Fix

The fix itself is simple: It worked pre-patch but not post patch. So we supply the affected binaries the old libraries without touching the main system. This means: A non-invasive, non update-breaking method. Steps as follows:

  1. We need the libraries that are patched just prior to Patch-22 from a running pre-Patch22,
  2. Placing those needed libraries (and only those) in a directory outside the scope of any update mechanism,
  3. Instructing the binaries to load these libraries instead,
  4. Testing the Setup.

Getting the libraries

The hard way is to install XenServer somewhere; you can even run it as a vm inside an already running XenServer. As you are reading this chances that you are running a XenServer somewhere are not entirely nil. Once done, Patch all the patches but do NOT install Patch 22. Copy these files to ‘Somewhere else’:

/lib/libssl3.so
/lib/libssl.so.6
/lib/libcrypto.so.0.9.8e
/lib/libssl.so.0.9.8e
/lib/UNINSTALL
/lib/libcrypto.so.6

Depending on your Version you need to replace /lib/ with /lib64/ if you run 6.5, Versions prior need the /lib/. Copy those files to your affected XenServer, like /root/crypt_fix.

Taming the services

On the affected server, stop all services that misbehave. In this example. bacula and puppet:

/etc/init.d/bacula-fd stop
/etc/init.d/puppet stop

Be sure the fixed files are in /root/crypt_fix (or any other directory to your liking, adjust the path to match yours):

total 1.9M
drwxr-xr-x 2 root root 4.0K Feb 18 13:27 .
drwx------ 6 root root 4.0K May 10 15:32 ..
-rwxr-xr-x 1 root root 1.4M Feb 18 13:05 libcrypto.so.0.9.8e
lrwxrwxrwx 1 root root   19 Apr 19 14:20 libcrypto.so.6 -> libcrypto.so.0.9.8e
-rwxr-xr-x 1 root root 235K Feb 18 13:20 libssl3.so
-rwxr-xr-x 1 root root 314K Feb 18 13:05 libssl.so.0.9.8e
lrwxrwxrwx 1 root root   16 Apr 19 14:20 libssl.so.6 -> libssl.so.0.9.8e

Edit /etc/sysconfig/bacula-fd to fix bacula (create if not existing):

# Lib Fix
export LD_LIBRARY_PATH=/root/crypt_fix/
LD_LIBRARY_PATH=/root/crypt_fix/

Repeat for any desired service (puppet, bareos..) and restart the services:

/etc/init.d/bacula-fd start
/etc/init.d/puppet start

The easy way

As I already did all the steps above packed the files into a tarball along with the example configs for bacula and puppet, download it here (x64) or here for x86 and install as follows:

tar -xv -C / -f /root/patch22_fix.tar.bz2
or 
tar -xv -C / -f /root/patch22_fix_x86.tar.bz2

Testing

Install lsof if not already installed:

yum install --enablerepo base,epel lsof -y

Now check that the services use the files from the fixed library dir:

lsof 2>/dev/null | egrep -i '(bacula-fd|puppet)' | egrep -i '(libcrypto|libssl)' | grep -v libk5
bareos-fd  5835     root  mem       REG                8,1      1365328     118088 /root/crypt_fix/libcrypto.so.0.9.8e
bareos-fd  5835     root  mem       REG                8,1       320712     118108 /root/crypt_fix/libssl.so.0.9.8efdfd

If you see that, it means it’s fixed! You can now rest easy that bacula/bareos/puppet/whatnot are running smoothly again! \o/

-Christian.

Christian

Touched base with Linux back in 1995, got hooked up on it ever since. I am using Linux for both private and office for two decades. Working as a System Administrator at a medium sized hosting company I get in touch with all kinds of trouble. All of which can be solved with Linux. In my blog I am sharing solutions to problems that I had to search for myself in hope that someone else out there might find them useful.

Leave a Reply

Your email address will not be published. Required fields are marked *