Datacenter move completed

Hey folks,

during the past 30 days we moved all the Hardware to a different data center (and provider). We tried to make the migration as seamless as possible but at some point everyone got hurt in the process.

Regarding IP ranges

Our new IP ranges are as follows:

IPv4: 84.246.124.132/27
IPv6: 2a01:1f8:2000::/48

Wait, IPv6? That’s right: We moved all servers to use IPv6 as primary address, with a fallback to IPv4 if needed. We also have some (minor) Servers running IPv6 only; but fear not, no production servers. You do not need to change anything on your end.

While on the matter of re-doing the entire thing I bought a new firewall, a WatchGuard XTM5 (http://www.guardsite.com/XTM-5-Series.asp). I upgraded the rather weak Celeron CPU to a Intel Core2Duo E5700 and increased the memory to 4gb. This is, for a firewall, beyond anything you can imagine. Its’ also fancy red, looking awfully sexy for a firewall (in my techie-opinion):

The firewall is running pfsense now, running (among other things) GeoIP-Block (No more Nigerian Spam), Snort (with subscription) and several other things. Even at full usage of the Up-link port the CPU of the firewall does not exceed 70%– and no packets dropped.

Beware the Dog! If your computer is doing an aggressive port scan or other nasty thing you will get blocked off the network for 1 hour. If you get bitten by accident, please contact me above (after the block is lifted) giving me your IP. I will look up the logs and adjust the firewall accordingly.

Regarding Power

I also picked up another Server, a used SuperMicro with 16 cpus and 24gb of ram. Not much, but the 4x4tb hard disks make a great Server to use for backups. For the most part it is identical with the main server: two power supplied, hardware raid doing RAID10, IPMI et all. In a worst case scenario this Server can run most of the important Servers as the main server does.

Regarding Backups

All your files, emails and whatnot hosted on this network are secured by hourly backups with Bareos, which data is stored on the other hardware. Daily Backups done and moved off-site and weekly dumps off all hardware to an encrypted hardware disk This means that if there is a catastrophic hardware failure the other server can (for the most part) pick up the load. If a plane crashes into the data center, the off-site data will still be available. If some freak software bug arises that wipes all the XenServers clean we have local attached backups and can get back online within a jiffy.
Bottom line: Your data is pretty much safe.

Sorry about the downtimes in the past 30 days. I hope your endurance paid off!
Christian.

Managing KernelCare with Puppet

KernelCare

If you haven’t felt it before: When Dirty Cow hit you did. The Linux Kernel is rock solid, proven but also has security issues. In this case: Root rights for everyone! And on top of that this bug is so trivially easy to exploit (several proof-of-concepts are out there that can easily converted into a life, working gun) that you had to update your kernels. On every server. And reboot.

The last part is especially evil because a reboot will be noticed by your customers if you are not employing some high-availability setup. And in the world of web hosting this is mostly not the case. So every reboot is a downtime, costs time and money. Plus, you have to update your servers in due time and plan said downtime accordingly. But for all this to happen your distribution must build and provide you with updates first. You can’t install non-existent patches.

Enter KernelCare.

KernelCare is a product from the folks that bring you CloudLinux, which solves all of the above problems. It consists of a kernel module that loads additional kernel patches for your kernel version and applies them in real time. The daemon checks for available updates every 4 hours (via cron) and patches are made available blazingly fast. To pick up the above Dirty Cow example, here is their incident reaction chart. To sum it up: You are days ahead. In a situation where remote root exploits is a thing, days can kill you.

Let’s rather kill the bugs.

Read More

Installing Vaultier under Fedora

Vaultering.

So there I was; trying to install Vaultier on a dedicated CentOS machine. Turns out there is only a docker installation, an installation script for Ubuntu and manual install. And for the latter it’s only for Ubuntu (or Debian). Tough luck.

But how hard can it be to install this in CentOS? Next to impossible. The software shipped in the default repositories (and epel) are too old to actually get it to work (without compiling a lot on my own, but that would break the nice updates). And updates are a must on a server that handles sensitive data.

So I took the next best thing: Fedora 24 Server. Even that turned out to be ugly; but in the end it worked. Here is how I did it.

Read More

XenServer, Patch 22 and Crypto

The What

I am using XenServer as my private solution for my network. It’s fast, reliable, open-source and free (as in free beer). I am sort of a fanboy. That said we are using XenServer at work, too.

Somewhat recently Citrix, maker of XenServer released hotfix XS65ESP1022 aka Patch 22, release notes:

This hotfix supports the upgrade of OpenSSL package to version 1.0.1.

Files Updated

stunnel-4.15-17.x86_64.rpm
make-3.81-3.el5.x86_64.rpm
openssl-wrapper-0.1-59.x86_64.rpm
openssl-xs-1.0.1e-42.xs15.x86_64.rpm
ca-certificates-2012.87-1.noarch.rpm
openssl-xs-libs-1.0.1e-42.xs15.x86_64.rpm
openvswitch-2.1.3-13.7579.x86_64.rpm
xenserver-transfer-vm-6.5.0-116122c.noarch.rpm

The bold one however, introduces some issues. If you (like everyone) installed extra packages in the Dom0 (the hypervisor) and maybe even used packages from epel then stuff will break apart. For example:

Read More

Authenticated with partial success

The What

It’s hardening time again.

Following up on my post “DNSSEC, SSH and keys.” this is another post in the series of hardening your SSH server and your server in general. Are you using password login or public keys?

1238895886504840879

Indeed. Why not both? And I am not going to recommend you that should put a password on your ssh keys (which is nice) but rather recommend real two-factor authentication: Public Key and a Password. What does it do?

Read More

Centos 6, PHP 5.6 and FastCGI

The What

So most of you are running (some sort of) a web-server. Mostly this is the usual LAMP stack, consisting out of

  • Linux
  • Apache
  • MySQL
  • PHP

and there is nothing wrong with that. Built on top of a Centos 6 machine yields a pretty fast and stable server. Yet, with the default setting and repositories of CentOS 6 this would yield with a mod_php run Apache Server featuring a whooping PHP 5.3. To remedy the version issue and switch to fastcgi yields in a modern set-up that’s lighting fast on top.

I will show you how to build a LAMP stack with stack apache, mod_fastcgi and a current PHP 5.6.

Let’s get down to business.

Read More

Upgrading Alfresco

The Situation

I am using Alfresco as a personal document archiving system for all my personal things. Everything that gets (snail-)mailed to me get scanned with my document scanner, which does OCR, page cutting, rotating and optimizing, and then uploaded to alfresco with pretty much one click. Well, one button (scanner), then I need to supply a file-name and finally the document is added via drag&drop to Alfresco. This system works great, even my wife uses it (frequently). The only thing that’s kept on paper are important documents like insurance policies and things like that.

As this is obviously a critical piece of our “digital lives”, it needs to be stable and somewhat up-to-date. I opt-out of going after every update and I seem to upgrade Alfresco only once a year, which works well for me. My Alfresco installation in shielded from the internet, that is, it’s on my private (non-internet connected) server, so bugs and exploits are of no concern here.

Now installing and running Alfresco is straightforward, consisting out of downloading, running and doing the setup and configuration wizard, period. But as some point, you do want to upgrade it. And this is where the fun starts.

Read More