The What
I am using XenServer as my private solution for my network. It’s fast, reliable, open-source and free (as in free beer). I am sort of a fanboy. That said we are using XenServer at work, too.
Somewhat recently Citrix, maker of XenServer released hotfix XS65ESP1022 aka Patch 22, release notes:
This hotfix supports the upgrade of OpenSSL package to version 1.0.1.
Files Updated
stunnel-4.15-17.x86_64.rpm make-3.81-3.el5.x86_64.rpm openssl-wrapper-0.1-59.x86_64.rpm openssl-xs-1.0.1e-42.xs15.x86_64.rpm ca-certificates-2012.87-1.noarch.rpm openssl-xs-libs-1.0.1e-42.xs15.x86_64.rpm openvswitch-2.1.3-13.7579.x86_64.rpm xenserver-transfer-vm-6.5.0-116122c.noarch.rpm
The bold one however, introduces some issues. If you (like everyone) installed extra packages in the Dom0 (the hypervisor) and maybe even used packages from epel then stuff will break apart. For example:
- Puppet (from epel, not puppetlabs repo) will be unable to create new certificates or segfault during a run,
- Bacula and Bareos will launch but crash on Data transfer. This is evil because a “status client” command always succeeds.
This is because the wrapper now points to newer OpenSSL libraries from the XenServer Patch; the binaries are however compiled against a lesser version wrecking all kinds of havoc. And since Citrix has the stance of saying “Don’t install stuff inside the Dom0” and Epel wont recompile against the libraries from Citrix this is now a stalemate without an escape. Let’s build one.
The Fix
The fix itself is simple: It worked pre-patch but not post patch. So we supply the affected binaries the old libraries without touching the main system. This means: A non-invasive, non update-breaking method. Steps as follows:
- We need the libraries that are patched just prior to Patch-22 from a running pre-Patch22,
- Placing those needed libraries (and only those) in a directory outside the scope of any update mechanism,
- Instructing the binaries to load these libraries instead,
- Testing the Setup.
Getting the libraries
The hard way is to install XenServer somewhere; you can even run it as a vm inside an already running XenServer. As you are reading this chances that you are running a XenServer somewhere are not entirely nil. Once done, Patch all the patches but do NOT install Patch 22. Copy these files to ‘Somewhere else’:
/lib/libssl3.so /lib/libssl.so.6 /lib/libcrypto.so.0.9.8e /lib/libssl.so.0.9.8e /lib/UNINSTALL /lib/libcrypto.so.6
Depending on your Version you need to replace /lib/ with /lib64/ if you run 6.5, Versions prior need the /lib/. Copy those files to your affected XenServer, like /root/crypt_fix.
Taming the services
On the affected server, stop all services that misbehave. In this example. bacula and puppet:
/etc/init.d/bacula-fd stop /etc/init.d/puppet stop
Be sure the fixed files are in /root/crypt_fix (or any other directory to your liking, adjust the path to match yours):
total 1.9M drwxr-xr-x 2 root root 4.0K Feb 18 13:27 . drwx------ 6 root root 4.0K May 10 15:32 .. -rwxr-xr-x 1 root root 1.4M Feb 18 13:05 libcrypto.so.0.9.8e lrwxrwxrwx 1 root root 19 Apr 19 14:20 libcrypto.so.6 -> libcrypto.so.0.9.8e -rwxr-xr-x 1 root root 235K Feb 18 13:20 libssl3.so -rwxr-xr-x 1 root root 314K Feb 18 13:05 libssl.so.0.9.8e lrwxrwxrwx 1 root root 16 Apr 19 14:20 libssl.so.6 -> libssl.so.0.9.8e
Edit /etc/sysconfig/bacula-fd to fix bacula (create if not existing):
# Lib Fix
export LD_LIBRARY_PATH=/root/crypt_fix/
LD_LIBRARY_PATH=/root/crypt_fix/
Repeat for any desired service (puppet, bareos..) and restart the services:
/etc/init.d/bacula-fd start /etc/init.d/puppet start
The easy way
As I already did all the steps above packed the files into a tarball along with the example configs for bacula and puppet, download it here (x64) or here for x86 and install as follows:
tar -xv -C / -f /root/patch22_fix.tar.bz2 or tar -xv -C / -f /root/patch22_fix_x86.tar.bz2
Testing
Install lsof if not already installed:
yum install --enablerepo base,epel lsof -y
Now check that the services use the files from the fixed library dir:
lsof 2>/dev/null | egrep -i '(bacula-fd|puppet)' | egrep -i '(libcrypto|libssl)' | grep -v libk5 bareos-fd 5835 root mem REG 8,1 1365328 118088 /root/crypt_fix/libcrypto.so.0.9.8e bareos-fd 5835 root mem REG 8,1 320712 118108 /root/crypt_fix/libssl.so.0.9.8efdfd
If you see that, it means it’s fixed! You can now rest easy that bacula/bareos/puppet/whatnot are running smoothly again! \o/
-Christian.
