pfsense on watchguard

Where there’s smoke…

As you folks have read I am using a WatchGuard XTM 5 Series as hardware firewall with pfsense. After running this in production for quite a while now I come to the conclusion that I can really, really recommend this setup. It is rock solid and working like a charm. This does not need to fear any commercial high end (and high cost) appliances out there.

Quick disclaimer: This howto is intended for the seasoned SysAdmin. There is no hand holding on installing pfsense or how to install or remove hardware. You’ll only get the pointers in this howto. I doubt beginners will buy and maintain hardware firewalls. Also this will void your warranty. But since the XTM5 are legacy production anyway you probably are not covered by any warranties anways.

Not only does this run pfsense without major modifications, it also is highly mod-able with lots of goodness. The default configuration of the XTM5 are as follows:

  • Celeron 440 2GHz CPU
  • 1GB of Ram
  • 1x 100Mbit Port
  • 6x 1gb Port
  • Onboard CF Card Slot

…There’s fire

By default the XTM boots from its internal CF Card that you could use to flash pfsense on it. But we do not want amateur level Firewall, we want a real trusted one. But before we get into modding the XTM, let me share some word of advice regarding what Model you want to get your hands on. You want a XTM 5 Series, period. The Series 5 comes in several models itself:

Read More

Datacenter move completed

Hey folks,

during the past 30 days we moved all the Hardware to a different data center (and provider). We tried to make the migration as seamless as possible but at some point everyone got hurt in the process.

Regarding IP ranges

Our new IP ranges are as follows:

IPv6: 2a01:1f8:2000::/48

Wait, IPv6? That’s right: We moved all servers to use IPv6 as primary address, with a fallback to IPv4 if needed. We also have some (minor) Servers running IPv6 only; but fear not, no production servers. You do not need to change anything on your end.

While on the matter of re-doing the entire thing I bought a new firewall, a WatchGuard XTM5 ( I upgraded the rather weak Celeron CPU to a Intel Core2Duo E5700 and increased the memory to 4gb. This is, for a firewall, beyond anything you can imagine. Its’ also fancy red, looking awfully sexy for a firewall (in my techie-opinion):

The firewall is running pfsense now, running (among other things) GeoIP-Block (No more Nigerian Spam), Snort (with subscription) and several other things. Even at full usage of the Up-link port the CPU of the firewall does not exceed 70%– and no packets dropped.

Beware the Dog! If your computer is doing an aggressive port scan or other nasty thing you will get blocked off the network for 1 hour. If you get bitten by accident, please contact me above (after the block is lifted) giving me your IP. I will look up the logs and adjust the firewall accordingly.

Regarding Power

I also picked up another Server, a used SuperMicro with 16 cpus and 24gb of ram. Not much, but the 4x4tb hard disks make a great Server to use for backups. For the most part it is identical with the main server: two power supplied, hardware raid doing RAID10, IPMI et all. In a worst case scenario this Server can run most of the important Servers as the main server does.

Regarding Backups

All your files, emails and whatnot hosted on this network are secured by hourly backups with Bareos, which data is stored on the other hardware. Daily Backups done and moved off-site and weekly dumps off all hardware to an encrypted hardware disk This means that if there is a catastrophic hardware failure the other server can (for the most part) pick up the load. If a plane crashes into the data center, the off-site data will still be available. If some freak software bug arises that wipes all the XenServers clean we have local attached backups and can get back online within a jiffy.
Bottom line: Your data is pretty much safe.

Sorry about the downtimes in the past 30 days. I hope your endurance paid off!

(Updated) Syncing your Data:

The Situation

After last weeks Amazons ban of rclone and acd_cli it was nigh impossible to access and sync your data reliable with pretty much anything. Sure, Amazon offers a free Windows client but that a) works only in Windows and b) does not make the drive available as a local drive. Plus rclone allows syncing of two clouds making them a de facto Raid1. It worked very well and due to the fact that it was scriptable you could fire-and-forget it. I never had any reason to look the other side.

Until now. I still don’t.

Read More

Server Status Page

Hello folks,

I will now publish all server related downtimes and information via a dedicated status page. Downtimes will no longer be published here. Visis (and bookmark) the new Server Status Page, available under or via the “Server Status” Menu on top of the website.

You can also subscribe to instant Email notifications here,
or subscribe to a real-time rss feed here.

If you have any issues with accessing the page, please do let me know.

Amazon Clouddrive, Encfs and Rsync – Part II

Welcome Back!

So now that you set up your own syncmachine that will push all your data encrypted to your Amazon Clouddrive, it’s time to up the ante. A quick recap: At this point you can mount and decrypt your Drive into a local folder and read files with ‘ok’ speeds. Writing actually takes ages because if you copy/move a file into your clouddrive decrypted folder the whole process of encrypting and uploading takes place. This is annoying for large files and some programs have serious problems with that.

So if you are a CouchPotato like me, that has some issues coming up on your Sonarr, err, Radar you want those issues to be gone. So like magic you raise your wand and begin to cast the “Abra-ca sabnzbd” spell.

– Anonymous post on usenet.

If that sentence made no sense to you, well no worries. Here is an analogy:

  1. You download you favorite Linux ISO with your favorite command line tool.
  2. You save said ISO in a temporary folder somewhere.
  3. You wrote or got a neat program that scans and sorts your temp folder into your CloudDrive.
  4. The program starts a copy/move from the temporary directory into the Clouddrive.
  5. Encfs and acd_cli start working on encrypting and uploading.
  6. Before the copy/move command completes your tool throws a timeout.
  7. Nothing happened.

Also if you share your CloudDrive folder with Samba it is annoying to transfer several GB worth of data. The wait kills at least me. I don’t like to wait.

Let’s fix all that. Now.

Read More

Managing KernelCare with Puppet


If you haven’t felt it before: When Dirty Cow hit you did. The Linux Kernel is rock solid, proven but also has security issues. In this case: Root rights for everyone! And on top of that this bug is so trivially easy to exploit (several proof-of-concepts are out there that can easily converted into a life, working gun) that you had to update your kernels. On every server. And reboot.

The last part is especially evil because a reboot will be noticed by your customers if you are not employing some high-availability setup. And in the world of web hosting this is mostly not the case. So every reboot is a downtime, costs time and money. Plus, you have to update your servers in due time and plan said downtime accordingly. But for all this to happen your distribution must build and provide you with updates first. You can’t install non-existent patches.

Enter KernelCare.

KernelCare is a product from the folks that bring you CloudLinux, which solves all of the above problems. It consists of a kernel module that loads additional kernel patches for your kernel version and applies them in real time. The daemon checks for available updates every 4 hours (via cron) and patches are made available blazingly fast. To pick up the above Dirty Cow example, here is their incident reaction chart. To sum it up: You are days ahead. In a situation where remote root exploits is a thing, days can kill you.

Let’s rather kill the bugs.

Read More

Amazon Clouddrive, Encfs and Rsync

Cloudy, with a chance of Unlimitness.

Amazon.$tld just released an unlimited Storage Tier for non-commercial applications for ~70 Bucks a year (depending on your location). And they are not kidding. Not only does the Web-Interface say ‘x used of unlimited’ but once mounted you’ll notice 100TB Quota. And reports on github actually state people reached that limit and even increased that by contacting support. So, plenty of space for your ‘documents’.

In its most basic form you can manage, down- and upload your files via the web-interface. That’s nice for on-the-go, but we are talking serious data here. So we need a client. Amazon offers the usual stuff: Windows Client, OSX Client. No Linux? Don’t be sad. Those clients can’t even do a real ‘sync’. Plus so far all clients upload your files unencrypted. Of course amazon goes ad-fishing with buzzwords from the ‘secure’-department… But they do have a list of compatible file types available. This is just for the web-client that it can index and play media; but this also means they are scanning your files (even if only automatically).

Err, no thanks.

What we want is the ability to mount the CloudDrive in Linux and encrypt everything before uploading.
Let’s get cracking.

Read More

Installing Vaultier under Fedora


So there I was; trying to install Vaultier on a dedicated CentOS machine. Turns out there is only a docker installation, an installation script for Ubuntu and manual install. And for the latter it’s only for Ubuntu (or Debian). Tough luck.

But how hard can it be to install this in CentOS? Next to impossible. The software shipped in the default repositories (and epel) are too old to actually get it to work (without compiling a lot on my own, but that would break the nice updates). And updates are a must on a server that handles sensitive data.

So I took the next best thing: Fedora 24 Server. Even that turned out to be ugly; but in the end it worked. Here is how I did it.

Read More

XenServer, Patch 22 and Crypto

The What

I am using XenServer as my private solution for my network. It’s fast, reliable, open-source and free (as in free beer). I am sort of a fanboy. That said we are using XenServer at work, too.

Somewhat recently Citrix, maker of XenServer released hotfix XS65ESP1022 aka Patch 22, release notes:

This hotfix supports the upgrade of OpenSSL package to version 1.0.1.

Files Updated


The bold one however, introduces some issues. If you (like everyone) installed extra packages in the Dom0 (the hypervisor) and maybe even used packages from epel then stuff will break apart. For example:

Read More