pfsense on watchguard

Where there’s smoke…

As you folks have read I am using a WatchGuard XTM 5 Series as hardware firewall with pfsense. After running this in production for quite a while now I come to the conclusion that I can really, really recommend this setup. It is rock solid and working like a charm. This does not need to fear any commercial high end (and high cost) appliances out there.

Quick disclaimer: This howto is intended for the seasoned SysAdmin. There is no hand holding on installing pfsense or how to install or remove hardware. You’ll only get the pointers in this howto. I doubt beginners will buy and maintain hardware firewalls. Also this will void your warranty. But since the XTM5 are legacy production anyway you probably are not covered by any warranties anways.

Not only does this run pfsense without major modifications, it also is highly mod-able with lots of goodness. The default configuration of the XTM5 are as follows:

  • Celeron 440 2GHz CPU
  • 1GB of Ram
  • 1x 100Mbit Port
  • 6x 1gb Port
  • Onboard CF Card Slot

…There’s fire

By default the XTM boots from its internal CF Card that you could use to flash pfsense on it. But we do not want amateur level Firewall, we want a real trusted one. But before we get into modding the XTM, let me share some word of advice regarding what Model you want to get your hands on. You want a XTM 5 Series, period. The Series 5 comes in several models itself:

Ranging from least-to-most expensive. The difference is not what you think; they all share the exact same hardware. The difference is with the software and license it comes with: The max throughput is limited as well as the maximum number of connections. You can easily upgrade between the series by purchasing a license- so the hardware remains untouched. See for yourself here: https://www.watchguard.com/.Β And since we will not be running the WatchGuard-OS anyway we are only looking at the hardware specs here. So go to eBay and pick up the cheapest XTM 505 you can find.
Protip: Some offerings on eBay come pre-tuned with upgrades CPUs. If you can, pick one of those.

While you are already shopping, also buy:

  • 2x 2GB DDR2 800MHz PC2-6300 DDR2 800 (240 PIN) DIMM 1.8v @~10 Bucks (used)
  • 1x Intel E5700 CPU @~5 Bucks (used)
  • 2x Any SSD you like, Any size will do.
  • 1x Cisco Console Cable USB to RJ45 @10 Bucks (new)
  • 2x 10 cm SATA-Cable @2 Bucks each (new)
  • Some Power strips (Those tiny duct-tape alike things that stick on both sides)

While the shopping list above is not required, it is very much recommended. It will more than triple the speed of your XTM, ready for prime time. Whatever that means to you.

We are aiming for:

  • 4 GB of Ram
  • Fast CPU that can easily handle a fully utilized 1Gb Port with snort using software bridges.
  • Boot from SSD using RAID1 (And making sure it boots from either)

Hardware

Once your shopping spree came to fruition, that is your hardware has arrived you can get your hands dirty. Well, not really- we are not car mechanics after all. Just unscrew the cover screws on the back One screw is behind the warranty label on the back (which you need to mind if you are a time traveler), some are on the sides. Simply unscrew and lift the cover open.

What you see then is pretty much standard PC issue: Motherboard with clearly obvious ram (plug yours in there), simply remove the old CPU and replace. Nothing really that should be explained to you. I doubt some non-technician will buy a hardware firewall and upgrade it. Oh, and unplug the CF card. Now the next part is more involved. The two SSDs and their cables can be placed with the Sata Cables on the right empty space (while looking at the front). But don’t go ahead and install the SSDs just yet. Just Upgrade CPU and Ram. Once that is firmly in place go ahead and power it on. It should come online as if nothing has changed. Once you confirmed that, read on.

Software

Now with pfsense we have a little of an hen-and-egg Problem. Let me run you through it. The BIOS of the XTM is locked by default, that is nearly none of the options can be changed and you would get the lovely ‘read only value’ message when trying to change anything. Among those read-only values is the boot-up device. So you can not run the installer from a stick.

Once pfsense can boot however you can flash the BIOS from within pfsense and unlock everything. But without an option to install… you get the point. Now there are several ways around this issue. You could write the installer image on the CF card (do you have a CF card reader available?). Or you could write the image on ‘the other’ SSD but that would make it impossible to set up a raid.

So I went the easy way of simply plugging the SSDs into my main PC and installing on there. In any case download pfsense from the website using these Download options:

  • File Type: Standard. The other option would be to Upgrade an installation, an option that I have never ever used.
  • Architecture: AMD64. We can go native 64bit with this, and 4gb scream 64bit.
  • Platform: USB Memstick Installer. You can go and download the CD if you want.
  • Console: VGA. Even tho the XTM has serial, this is for the installation. And we will be installing on a pc.

Once you have your installation file, flash it on an USB Stick or burn the CD. Turn off your pc, unplug your current SSDs and HDDs, install your newly bought SSDs (both) and plug in your USB stick. Turn it on again and boot from the stick. Let it boot as usual. After selecting your keyboard layout you are represented with the ‘Select Task’ Menu. Select ‘Setup GEOM Mirror’ here. Select ad0 first, ada1 second. Confirm and once you are back in the ‘Select Task’ Menu, pick Quick & Easy Setup. In the last step pick standard kernel. Poweroff after installation.

So now you have two SSDs, mirrored, ready. Now install those into the watchguard using the tiny SATA cables you purchased. Fixate the SSD with the power strips on the side. Re-attach the cover. If everything went well and you followed the instructions you now have a Watchguard with upgraded CPU, 4gb ram and Dual SSD in a RAID1 installed.

A word on Upgrading

The difference between the embedded and standard version are considerable. For one /tmp and /var are ramdisks that are volatile and not kept across reboots. An exclusion to this are RRD graphs which are copied over to the hard disk/ flash/ SSD. This behavior can also be replicated by ticking the appropiate boxes in the administration interface in the advanced section (one reboot required). This is recommended to lessen the writes on the SSDs.
The other (more important) issue is that every upgrade is a full reinstall with a restoration of the configuration. This means the boot medium is separated into two partitions and an upgrade would wipe the currently non-active one empty, install the new pfsense there and restore the current running config in there. The next boot will be from that partition. If something fails you can simply boot from the older partition again. Which this sounds neat this will also wipe all customizations you might have done. A Full installation is like a normal OS installation, files are updated in place using the administration interface. I really recommend the standard installation. It’s upgrade-safe, can do GEOM-Raid and keeps all your precious fine tunings.

First Bootup

Attach the Console Cable to your PC (not the firewall) and set it up. “Set it up” mostly consists out of plugging the USB end in the USB Port. Fire up cutecom, putty of whatever your choice of a serial console client is. Try to connect- you should see pretty much nothing (but connection is established). Now attach the cable to the WatchGuard Box and power it on. You should see Bootup BIOS messages which will cease after the POST messages. Background: The locked Bios can not handle console redirection (or does not allow) post BIOS. Once you confirmed your XTM is still operational, unplug the cable (both ends) and re-attach. Now you should see pfsense booting.

Go through the initial setup on the console, don’t bother being accurate. That comes later. For now, get to the main menu and hit ‘8’ for Console. Enter ‘dmidecode | less’ At the very beginning and find this block:

BIOS Information
 Vendor: American Megatrends Inc.
 Version: 080015 
 Release Date: 02/03/2010 
 Address: 0xF0000 
 Runtime Size: 64 kB 
 ROM Size: 1024 kB

The release date must be exactly ‘02/03/2010′. Only then do you have a flash-able Bios, which is almost always the case. If you confirmed that, install the package flashroom by typing

pkg install flashrom
rehash
fetch https://misc.alpha-labs.net/pfsense/xtm5_83.rom

This will only install the tool required, rehash makes new binaries to be found by the shell and the fetch actually fetches the new ‘fixed’ BIOS. I signed the BIOS with my GnuPG key, get the verification here. Grab all your courage and flash your BIOS with:

flashrom -p internal -r backup.rom
flashrom -p internal -w xtm5_83.rom

The first command backups your current BIOS, If all went well, exit the console and power off the XTM. Unplug all cables and let it discharge all remaining power for 30 seconds. Open up the cover again and remove the battery. This is important to reset the BIOS settings to the (new and unlocked) defaults, removing the read-only variables. Do note the position of the battery in regards of +/- poles. Re-attach and power it back on. This time, hit tabulator during BIOS screen and you should end up inside the BIOS. Fix things to your hearts desire. Suggested at least is to set the SATA Ports to AHCI mode. If you set the serial console redirection to “always” you can now keep the serial cable attached during boot.

Power it on and let it boot. You should be able to see the boot process from BIOS all the way to the pfsense menu.

WatchGuard Configuration

Once you are at this point, follow the pfsense first time setup as you would normally. Get it up and running so you are at some point logged in at the dashboard. Go to ‘SystemPackage > Manage Installed > Packages’ and install:

LCDproc

LCDproc makes the little LCD screen of the WatchGuard actually work. Once installed, to go ‘Services > LCDproc > Server’ and set it up like this:

  • Com Port: Parallel Port 1
  • Display Size: 2 rows 20 columns
  • Driver: WatchGuard FireBox with SDEC
  • Port Speed: Default

The rest is to your liking. I like to see the current throughput and system load as well as the hostname in 5 second intervals.

Backing up the configuration

As this will be a vital part of your infrastructure, you want to be able to recover as fast as possible,so you need the current configuration of this pfsense. Luckily, this is easy achieved. Create a new user inside the pfsense administration interface and give that user clearance for SSH Access. On a different (any) server, create ssh-keys for a non-root user and add those keys to the user you just created in pfsense. Verify that you can login from the server into the pfsense with that username.

The configuration for the running pfsense is kept at '/conf/config.xml', so it's a trivial matter of copying the file over onto the server. I am using this tiny script which you can use as a starting point:

#! /bin/bash
REMOTE_IP="10.6.0.1"
REMOTE_NAME="yourfirewall.fqdn.com"
USER="remote-user"

cd "$(dirname $0)"
PATH="$(pwd)"

function error () {
 echo "Error: $*"
 exit 2
}

test -e ${PATH} || mkdir -p ${PATH} || error "Unable to create ${PATH}."
cd ${PATH} || error "Unable to cd into ${PATH}."
/usr/bin/git reset --hard 
/usr/bin/git pull
/usr/bin/ssh ${USER}@${REMOTE_IP} cat /conf/config.xml > "${PATH}/${REMOTE_NAME}.xml" || error "Unable to get config."
/usr/bin/git commit "${REMOTE_NAME}.xml" -m "Automatic update." || error "Unable to git-commit."
/usr/bin/git push || error "Unable to git-push."

As you can see I am also pushing the configuration straight into git. You can omit that or adapt your configuration to match. In any case I recommend having an offsite backup of your configuration in case of total failure. You will love the backup if you can skip re-doing hundreds of rules by hand.

Monitoring

If you run a serious setup, that is, you run icinga or nagios you really want to monitor your firewall. So install the NRPEv2 package from the list of available packages and set it up (services -> nrpev2). Enable nrpe on the top of the page, set your Nagios Server IP and tick ‘allow Arguments’. Set up the commands as desired.

I am using smart and geom (raid) check in addition to the stock checks. Simply download these files:

and place them on the pfsense under ”/usr/local/libexec/nagios’. Once there, add the corresponding checks (after refreshing the page); gmirror check does not need any arguments- check_smart does. Add ‘-d /dev/ad8’ for example as one argument. For our two SSDs add two checks. Do not add two arguments to the one check, it will fail.

Also check (activate) the ‘use sudo’ button for check_smartmon. Add this line to ‘/usr/local/etc/sudoers’:

nagios ALL= NOPASSWD:/usr/local/libexec/nagios/check_smartmon

This will allow execution of this check as root.

Conclusion

This is how I set up my WatchGuard XTM 5 for production use, along with NRPEv2 for monitoring (Monitoring the Smart status of the SSDs as well as the GEOM Raid Information). I hope you were able to follow my instructions and have a blast of a time with your new, awesome production grade firewall for a penny.

-Christian.


Further reading:

https://doc.pfsense.org/index.php/PfSense_on_Watchguard_Firebox#XTM_5
http://www.triebwerk23.de/joomla/index.php/firewalls/watchguard-xtm-5-xtm-505-515-525-545-pfsense-64-bit
https://forum.pfsense.org/index.php?topic=43574.0

Christian

Touched base with Linux back in 1995, got hooked up on it ever since. I am using Linux for both private and office for two decades. Working as a System Administrator at a medium sized hosting company I get in touch with all kinds of trouble. All of which can be solved with Linux. In my blog I am sharing solutions to problems that I had to search for myself in hope that someone else out there might find them useful.

7 thoughts to “pfsense on watchguard”

  1. It looks like a really good solution, but it’s not future proof, because the CPU is missing AES-NI, which will be a requirement for future versions of pfSense. Sorry πŸ™

    1. Hey same-name-as-me,

      You are of course right (https://www.netgate.com/blog/pfsense-2-5-and-aes-ni.html) but since 2.3 will be maintained for at least a year after the release of 2.4 which itself will be release in at least a year from now you have some years off maintenance left. Plus even after EOL you can keep using the firewall (if a port is closed, it’s closed.).

      There are also other options (providing a different kernel that does not require aes-sni) or maybe there will be a fork or workaround. In worst case you / we can switch to a different OS altogether.

      I also checked: There are no CPUs for socket 775 that support aes-sni.
      Summed up: Cheap yes, some years left: check. Everything after that: we will see πŸ™‚

      -Chris.

      1. I switched my Watchguard box from pfSense to OPNsense to head off this problem. You can (mostly) load a pfSense config backup file in OPNsense to transfer your settings.

        Also, regarding the article itself: I found it easier to write the installer to a CF card than to mess around with BIOS and plugging-unplugging cables. Worked for me with both pfSense and OPNsense. Remove the CF card after installation.

    1. Hey aWatchfuleye,

      I gave credit to the sources I used to research the post. During which I did not stumble across said post; similarities might occur as the steps should be the same in every other related howto or post.

      Nevertheless I added the Link to the further reading/credits box. I will also continue to add more links if I see them.

      Thanks for the pointer!
      -Chris.

  2. Thanks for the info and your work!
    Is it possible to modify the xtm5_83.rom file? I would like, for example, to change the ‘pfSense’ messages on the LCD to ‘OPNsense’. πŸ™‚ Any hints on how to unpack / re-pack the rom file, CRCs, etc would be appreciated.
    Thanks,
    John

Leave a Reply

Your email address will not be published. Required fields are marked *