PFsense and XenServer

The players

I like XenServer. It’s a rock solid (albeit basic) virtualisation platform that’s not only open-source but can handle any OS you throw at it. Management is a bliss and in my many years of using it both professionally and privately I yet have to encounter a (non-hardware related) crash or other issue with it.

Installation is a breeze. All you need is the ISO, burn it an install it on some hardware you have lying around (it works even inside VirtualBox for a Test-drive; and yes: It also works inside XenServer. Xenception.). The hardware specifications are based on what you are going to do with it. From basic testing to high end computing with several hundreds of cores– no problem there.

The other piece of software I totally like is pfsense, a software based firewall distribution. With some minor tweaking you can get a real neat setup working.

Those two are just screaming to get together and have  party. Bring the party hats!

The old setup

Before we head on with the issue and the fix let me explain you my setup. I used to run several VMs directly connected to the Internet, having both a public ip and a local firewall. I also had a second internal-only virtual lan that was used for backup and secure inter-vm communication as this virtual lan is never routed outside the XenServer host. XenServer has the ability to create a ‘server-only lan’, a virtual network switch which traffic can never be routed on the outside ‘real’ interface. It worked well, but it was a hassle. I had to monitor and manage the firewall rules on every public VM and also manage and maintain a list of private IPs in my LAN.

Pfsense changed that. The firewall is now the only VM that is directly connected to the Internet and having all the public ips and firewall rules. All my VMs now only have one internal IP and are all solely in my ‘server-only lan’; forcing all traffic through the firewall. No need to manage two ips or firewall rules on the VMs itself. The firewall does a 1:1 nat with a default-to-deny rule.
A real life example: My Nameserver ns1.alpha-labs.net has the address 46.229.47.133, which is active on the firewall only. The Nameserver VM has no clue about this ip, only it’s private one:

[chris@ns1 ~]$ ip a s
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 76:84:6d:ee:33:78 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.3/24 brd 10.1.0.255 scope global eth0

The firewall would pass on any connections coming in on 46.229.47.133 to the internal ip 10.1.0.3 if the firewall has an allow rule for this, else it is silently dropped. The fact that there will be no packets coming in from the firewall that are not allowed negates the need for a local VM firewall (I still have them, tho.).

You can of course switch to a transparent firewall, managing no ips on the firewall itself. This is a neat approach if you want one central place to block dos or do some traffic shaping.

The issue is not a lack of appliances; the issue is getting pfsense to work in XenServer.

Making it work

Installation itself is pretty straightforward. Plug the pfsense installation CD into the virtual drive, choose “Other installation media” from the new VM wizard and go ahead and install it. When prompted, finish the installation and configuration wizard so that you finally can log in to the web configuration.

In theory you can now go ahead and have fun with your new pfsense firewall. However you will soon notice that no matter what you do you can not reach any host behind the firewall, inside the server-only lan. This is a problem with XEN an how it simulates network interfaces. In incorrect lamens terms: If a VM has two interfaces and you want to receive in from one interface and simultaneously transmit on the other (as is the case with a transparent or nat firewall, read: always) both network interfaces block each other, only one is allowed to do some sort of action. This results in the ability to ping and even for very slow network connections but all will eventually time out. Debugging this is was hell.

The fix is two-fold. For one you have to disable all hardware related offloads inside the pfsense. Head over to the advanced setup and in there networking. Tick the following boxes:

pfsense-settingspfsense will ask you to reboot; instead to a full power-off. Like I said; two fold. The Other part: Once the machine is halted we need to disable the same on the xenserver side. This can only happen if the VM is powered off completely.

Log in via ssh to your xenserver, become root (if not already).

Then, either copy the VM-UUID from XenCenter (VM -> General, last row. You can even copy&paste the uuid.) or check the vm-uuid via this command:

xe vm-list

uuid ( RO)           : f6874e6c-656b-9a1a-8ec3-0dcc0972a6db
     name-label ( RW): infra - fw1
    power-state ( RO): running

the “f68[…]6db” is the UUID. Now let’s get all the network cards attached to this VM:

xe vm-vif-list uuid=f6874e6c-656b-9a1a-8ec3-0dcc0972a6db

uuid ( RO)                  : c4f73f9c-595f-e911-07d2-10d81490386f
         vm-name-label ( RO): infra - fw1
                device ( RO): 0
                   MAC ( RO): ce:3b:cc:7a:9b:df
          network-uuid ( RO): e6e2d955-4e2a-ea80-0114-905f2fa19b13
    network-name-label ( RO): Internet


uuid ( RO)                  : b337d209-c17e-6a59-0376-1512da0465f0
         vm-name-label ( RO): infra - fw1
                device ( RO): 1
                   MAC ( RO): 4a:e8:98:02:86:c3
          network-uuid ( RO): 6febe039-144c-0b61-018d-378def26a5f3
    network-name-label ( RO): pfsense

Of course, replace the uuid with yours. In this case there are two network card emulated. We need to disable several network related flags for each card.

VIF="c4f73f9c-595f-e911-07d2-10d81490386f"
xe vif-param-set uuid=${VIF} other-config:ethtool-gso="off"
xe vif-param-set uuid=${VIF} other-config:ethtool-ufo="off"
xe vif-param-set uuid=${VIF} other-config:ethtool-tso="off"
xe vif-param-set uuid=${VIF} other-config:ethtool-sg="off"
xe vif-param-set uuid=${VIF} other-config:ethtool-tx="off"
xe vif-param-set uuid=${VIF} other-config:ethtool-rx="off"

Repeat this for the second card as well: Change the first line with VIF and replace the UUID with the second network cards VIF and re-run all following lines. On success you will just be returned to the shell, no output of any kind.

That’s it. On the next pfsense VM start you can now reach all VMs behind it.

Tracking changes

A small trick: pfsense has it’s current configuration as an xml file sitting in /conf/config.xml. Said file can also be used as a disaster recover by “restoring” from the web configuration. A freshly installed pfsense can be fed that xml file and a reboot later your firewall is alive and kicking again with all the bell and whistles; minus the rrd graphs. You can, however, back them up, too.

I wrote this small, very basic script that runs every 10 minutes and pushes the changes into my git. No more fatal screw-ups. Well, there might be, but reverting is a breeze:

#! /bin/bash

REMOTE="fw1.int.alpha-labs.net"
PATH="/home/backup/fw1-config"
USER="backup"

function error () {
        echo "Error: $*"
        exit 2
}

test -e ${PATH} || mkdir -p ${PATH} || error "Unable to create ${PATH}."
cd ${PATH} || error "Unable to cd into ${PATH}."
/usr/bin/git reset --hard 
/usr/bin/git pull
/usr/bin/ssh ${USER}@${REMOTE} cat /conf/config.xml > "${PATH}/${REMOTE}.xml" || error "Unable to get config."
/usr/bin/git commit "${REMOTE}.xml" -m "Automatic update." || error "Unable to git-commit."
/usr/bin/git push || error "Unable to git-push."

The error function an easily be made to report to your local icinga/nagios or email you if there is any kind of trouble. And since the xml file is pure textfile you can easily see what change broke your neck later on.
Enjoy your new awesome, virtualised firewall!

-Chris.

Christian

Touched base with Linux back in 1995, got hooked up on it ever since. I am using Linux for both private and office for two decades. Working as a System Administrator at a medium sized hosting company I get in touch with all kinds of trouble. All of which can be solved with Linux. In my blog I am sharing solutions to problems that I had to search for myself in hope that someone else out there might find them useful.

3 thoughts to “PFsense and XenServer”

  1. Hi Christian
    If I may … and this seems like the closest solution to my problem.

    I have an LTE 4G device (FritzBox – my internet), it is also wireless capabe and has 4x 1GB LAN NICs. One NIC is connected to XenServer (with pfSense running as a VM). pfSense’s WAN interface is this connection (as per XenCentre config of the NICs assigned to the VM). pfS’s LAN NIC is one of the other physical NICs on the server connected to a D-Link Websmart switch, manageable [VLANS etc]. The server is a Supermicro, 12x NICs, ample CPU’s, HDDs & RAM). XenServer itself has no vSwitch configs (but I have a feeling I have to do that…)

    I have been wrecking my brain to get OpenVPN to work so that I can remote into the LAN and do remote tasks. OpenVPN is straight forward install (no fancy routed networks yet). I just want to connect at this stage and be able to open pfS’s Management interface, connect to XenServer using XenCentre, remotely install some other VMs, do normal network and installation tasks, there are 4 Linux PCs etc, etc… (trying to start an Open Source company and this is my LAB, demo as well as Live environment…)

    XenServer 6.5 with latest service packs and patches and pfSense, latest version; both 64bit…

    I did find out eventually that the ISP is blocking VPN & Dynamic DNS services on the LTE network. I had that changed; I now have static IP and DDNS is working.

    What happens when I OpenVPN (from LX Mint) I connect perfectly to the VPN service; I get my IP as defined by OVPN and or OVPN Client Specific overrides, gateways, dns servers etc.). I can ping the IPs on the XenServer Management LAN, switches, pfSense itself. I can even ssh into Xen & pfS, but I can only issue short commands, like “ls /” or other proper commands etc. If I issue stuff like TOP or xsconsole (seems like anything that needs to refresh the screen), something happens, but I get no further reply (i.e., I see that I pressed enter, but now nothing happens; it freezes). I cannot open pfS web management; Firefox says “looking up host, found host, connecting, transferring” and then it stays with transferring data, but nothing happens. XenCentre just times out eventually trying to connect to the server.

    Then I started thinking if it might have something to do with routing (specifically XenServer’s default gateway on the MGT LAN… then I came across this post of yours…

    Pretty nice work you’re doing here… 🙂

    Anyhoo, I was just wondering if you can maybe shed some more light for me on this matter or spare some advice. I have a feeling this post of yours is kinda the start to my solution…

    Hope to hear from you soon. Thank you in advance.

    Charl

Leave a Reply

Your email address will not be published. Required fields are marked *