This is going to be a bit of a long post, so I’ll be brief in the intro.
Over the last few years, I’ve become enamored with routing. As part of that, I’ve spent a lot of time exploring the various virtual routing platforms, trying to eek out the best performance possible. So I decided to put them all head-to-head in a virtual setting and see how they fare.
The victims:
- pfSense 2.4.4-p3
- OPNSense 19.7
- VyOS 1.2.2
- Mikrotik Cloud Hosted Router 6.45.3, unlimited license (CHR)
- Just basic Debian Buster running FRR
- All tests were done with
iperf3 -c IP -P2. I tried a number of different combinations of window sizes, number of streams, etc, and everything was pretty close to equal. UDP obviously gave much bigger numbers, but I’m interested in TCP for right now.
For the purpose of testing and to make all things equal, pfSense and OPNSense will be running with NAT disabled and pfctl -d, to remove the potential slow-downs from packet inspection. The CHR, Debian, VyOS installs won’t have firewalls running.
If you are just interested in the final results, they are HERE. Or maybe you’d like the results when I ran the test on ESXi HERE.
The Layout
They say a picture tells a thousand words, so I think a simple network diagram should illustrate the setup:
I’ll be replacing the firewall image in this picture with the various victims.
I’ve got a LOT of routing going on in my network. At last count, I think I have at least 8 hops from one end of my network to the other, so OSPF simplifies adding new subnets behind various routers. In most cases, I don’t even bother to add default routes to new hosts as OSPF floods the default route from my edge device.
The Hardware
For these tests, I wanted to use KVM. The network stack on ESXi has been driving me crazy lately, so I was hoping KVM would give me some decent results. Proxmox makes this pretty trivial, so that’s what I set up.
- Proxmox 6
- rootonzfs on an SSD
- 64GB of RAM
- D-1521, with Dual X550/552 10G-BaseT ports
This is a more-than-capable machine for these tests, though I will say I’ve gotten better results on my Broadwell E5.
The networking config on the host is pretty simple. To further isolate things, I have two different ports running as Trunks. Proxmox networking goes over one, and the networking for this test lab goes out of the other.
I did try using OVS, just to see if the results were any different, but they weren’t. So on Proxmox, the network config is pretty simple:
auto lo
iface lo inet loopback
iface eno4 inet manual
auto ens2
iface eno2 inet manual
auto eno3
iface eno3 inet manual
auto vmbr0.3
iface vmbr0.3 inet static
address 10.3.1.84/24
gateway 10.3.1.1
auto vmbr0
iface vmbr0 inet manual
bridge_ports eno3
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes
auto vmbr1
iface vmbr1 inet manual
bridge_ports ens2
bridge_stp off
bridge_fd 0
bridge_vlan_aware yesAll VMs are connected to vmbr1.
The VMs should be fairly equal. CPU-wise, I’ve gone with 2c2t. RAM is 2GB, which is overkill for basically any of these
The Linux crowd
The first batch of routers are all based on Linux. The VyOS version I used (1.2.2), is based on Debian Jessie with a modern 4.19 kernel and runs FRR. The CHR is based on an ancient 3.x Linux kernel. The plain Linux install will be the latest Debian Buster, with the repo version of FRR, 6.0.1.
VyOS
As a contributor, I have access to the current VyOS crux version, 1.2.2. For routing, it uses FRR 7.0.1.
config
The config is painfully simple. Three interfaces, WAN/LAN/LAN2, OSPF, DHCP on the LAN interfaces (for ease of use) and enabling SSH.
As initially mentioned, there’s been no firewall configured here. I’m only interested in pure routing performance.
interfaces {
ethernet eth0 {
duplex auto
smp-affinity auto
speed auto
vif 500 {
address 10.253.253.2/30
}
vif 2222 {
address 10.222.222.1/24
}
vif 2223 {
address 10.223.223.1/24
}
}
loopback lo {
}
}
protocols {
ospf {
area 0.0.0.0 {
network 10.253.253.0/30
}
redistribute {
connected {
metric-type 2
}
}
}
}
service {
dhcp-server {
shared-network-name VLAN2222 {
subnet 10.222.222.0/24 {
default-router 10.222.222.1
dns-server 10.53.53.53
range 0 {
start 10.222.222.10
stop 10.222.222.200
}
}
shared-network-name VLAN2223 {
subnet 10.223.223.0/24 {
default-router 10.223.223.1
dns-server 10.53.53.53
range 0 {
start 10.223.223.10
stop 10.223.223.200
}
}
}
}
ssh {
port 22
}
}boot time
VyOS generally boots pretty quickly, at least with a simple config. 37 seconds from starting the VM to routing packets. That’s also with the bootloader timeout.
iperf3 results
Here are the results of a simple iperf3 test. From 10.223.223.11 to 10.222.222.12 :
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 7.17 GBytes 6.14 Gbits/sec receiver
[ 8] 0.00-10.04 sec 7.22 GBytes 6.18 Gbits/sec receiver
[SUM] 0.00-10.04 sec 14.4 GBytes 12.3 Gbits/sec receiverI would be lying if I said I wasn’t at least a little bit disappointed with those results. Especially since the direct VM->VM on the same layer2 test shows quite a bit more performance:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 11.8 GBytes 10.1 Gbits/sec receiver
[ 8] 0.00-10.04 sec 11.7 GBytes 10.0 Gbits/sec receiver
[SUM] 0.00-10.04 sec 23.5 GBytes 20.1 Gbits/sec receiverI guess it’s just the hardware.
I have a few VyOS VMs on ESXi as my main intervlan routing devices on bigger hardware, and those are capable of 15-30Gbps routing between VMs on the same host depending on how busy the host is.
speedtest
I have gigabit Internet, so I wanted to make sure results are at least close to what I would get natively.
Given that this lab is behind a few routers, and my network is fairly busy at any point in time, these results are close enough to gigabit to count.
CPU usage
I was curious what kind of CPU usage there would be under max routing load. Obviously adding a firewall would add to this a bit:
Debian/FRR
The was just a plain Debian Buster install with FRR installed. Honestly, most of the results are throwaway, since they are largely identical to VyOS.
config
There isn’t much in the way of special config here. Enabling net.ipv4.ip_forward = 1 in /etc/sysctl.conf is a must.
Networking ( /etc/network/interfaces ):
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto ens18
iface ens18
auto vlan500
iface vlan500
bridge-ports ens18.500
bridge-stp on
address 10.253.253.2/30
gateway 10.253.253.1
dns-nameservers 10.53.53.53
auto vlan2222
iface vlan2222
bridge-ports ens18.2222
bridge-stp on
address 10.222.222.1/24
auto vlan2223
iface vlan2223
bridge-ports ens18.2223
bridge-stp on
address 10.223.223.1/24FRR ( /etc/frr/frr.conf ):
frr version 6.0.2
frr defaults traditional
hostname ovs
log syslog informational
service integrated-vtysh-config
!
router ospf
redistribute connected
network 10.253.253.0/30 area 0
!
line vtyisc-dhcpd ( /etc/dhcpd.conf ):
option domain-name-servers 10.53.53.53;
default-lease-time 600;
max-lease-time 7200;
subnet 10.222.222.0 netmask 255.255.255.0 {
range 10.222.222.10 10.222.222.200;
option routers 10.222.222.1;
}
subnet 10.223.223.0 netmask 255.255.255.0 {
range 10.223.223.10 10.223.223.200;
option routers 10.223.223.1;
}boot time
This is the one metric that isn’t throwaway compared to VyOS. 19 seconds from power-on to routing is FAST, and that’s with 5 seconds of grub time. This means if you are interested in super fast boot times, it’s probably worth your time learning how to set up a router/firewall on unadulterated Linux.
iperf3 results
As hinted at, the results are very similar to the VyOS results. Not surprising considering same kernel versions, both are FRR, etc.
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 7.14 GBytes 6.11 Gbits/sec receiver
[ 8] 0.00-10.04 sec 7.08 GBytes 6.06 Gbits/sec receiver
[SUM] 0.00-10.04 sec 14.2 GBytes 12.2 Gbits/sec receiverspeedtest
With these speedtest results, I must have found a rare moment when the rest of my network was mostly idle:
CPU usage during heavy routing
Again, similar and expected results:
All in all, the differences between FRR and VyOS were pretty minimal. Despite the longer boot times, I would probably just use VyOS, for the CLI and ease of configuration.
CHR
I’m a heavy Mikrotik user otherwise (multiple RB4011s, CRS317, 326, hex, haps, etc), and CHRs are something I feel like should really perform well.
This is where things started to get a bit funky. I’ll be honest here, the CHR results were the ones I was most interested in. I have NEVER been able to get decent routing results out of a CHR, I was hoping the proverbial “blank slate” would allow me to get the real picture.
The VM was a freshly installed (via systemrescuecd), 6.45.3 CHR, unlocked with a purchased unlimited license.
config
The config here was painfully simple again. A few interfaces and DHCP servers.
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no name=Trunk
/interface vlan
add interface=Trunk name=LAN vlan-id=2222
add interface=Trunk name=LAN2 vlan-id=2223
add interface=Trunk name=WAN vlan-id=500
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=10.222.222.2-10.222.222.254
add name=dhcp_pool1 ranges=10.223.223.2-10.223.223.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=LAN name=dhcp1
add address-pool=dhcp_pool1 disabled=no interface=LAN2 name=dhcp2
/routing ospf instance
set [ find default=yes ] redistribute-connected=as-type-2 router-id=10.253.253.2
/ip address
add address=10.253.253.2/30 interface=WAN network=10.253.253.0
add address=10.222.222.1/24 interface=LAN network=10.222.222.0
add address=10.223.223.1/24 interface=LAN2 network=10.223.223.0
/ip dhcp-server network
add address=10.222.222.0/24 gateway=10.222.222.1
add address=10.223.223.0/24 gateway=10.223.223.1
/ip dns
set servers=10.53.53.53
/routing ospf network
add area=backbone network=10.253.253.0/30
/system clock
set time-zone-name=America/Chicagoboot time
Frankly, the boot time was really about the only decent thing about my CHR tests. 12 seconds, which is in-line with the other Linux results, there’s just no 5 second timeout in grub to contend with here.
iperf3 results
These results really made me question my testing methodology. Abysmal is about all I can say about it:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 1.84 GBytes 1.58 Gbits/sec receiver
[ 8] 0.00-10.04 sec 1.67 GBytes 1.43 Gbits/sec receiver
[SUM] 0.00-10.04 sec 3.51 GBytes 3.01 Gbits/sec receiverI really spent a lot of time fighting these results. They felt way off. Some of the troubleshooting:
- Messing with offload settings, both in the host and in the VM.
- Using different NIC types. Maybe the CHR didn’t like virtio.
- Using a Mellanox vs an Intel NIC. The results were the same whether between the same bridge on Proxmox, to another physical host over the 10Gb Trunk link, etc.
- Changing around all the VM’s involved CPU cores, RAM, NIC types, anything else I could think of.
In the end, I couldn’t do anything to really change the results. They held pretty steady at 3Gbps.
speedtest
As mentioned above, probably a throwaway test. I just happened to run this test when my network was a little bit busier.
cpu usage
Interestingly enough, the CPU usage on the CHR is quite a bit higher, despite also Linux. I’m guessing this is attributed to the fact that quite a few networking enhancements and optimizations have occurred in later Linux kernel versions + FRR.
I was REALLY disappointed with the Mikrotik results here. I know RouterOS is more than capable of going faster than those speeds, but I’m not sure what kind of tweaking it takes to get it. Maybe you need lots of sources and destinations.
I know my RB4011 can do almost full 10Gb (unless you are on IPv6 grumble grumble), CCRs can do way more, so I don’t know why this thing is so slow. And it’s been like that every time I’ve tested it, whether on KVM or ESXi.
FreeBSD
It definitely wouldn’t be a complete test without also testing two popular players in the FreeBSD world. OPNSense and pfSense.
For these tests, a number of important configuration options were applied:
- All hardware offload stuff disabled in the VMs.
- Three VLAN interfaces, WAN/LAN/LAN2 as mapped out above.
- FRR plugin installed.
- NAT completely disabled.
pfctl -d. While it didn’t really impact these tests, I wanted to make sure it was a level playing field.
pfSense
I’ll be honest. I think these *Sense platforms aren’t really great routers. Can they route intervlan? Sure, but even with FRR somewhat native now, it’s just a pain and very clunky.
With pfSense, I never actually did get FRR running. It starts, but it just doesn’t do anything. Not to mention the GUI for it absolutely sucks and is just a confusing mess. It doesn’t actually impact these tests, but I annoyingly had to create some static routes on my other routers to get everything working.
boot time
Not fast, 37 seconds with a few seconds at the bootloader. I’m also positive I’ve seen this take considerably longer when it’s not a basic config.
iperf3 results
Again these results made me question my testing methodogy a bit:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 1.04 GBytes 891 Mbits/sec receiver
[ 8] 0.00-10.04 sec 1.97 GBytes 1.68 Gbits/sec receiver
[SUM] 0.00-10.04 sec 3.01 GBytes 2.57 Gbits/sec receiver It’s worth noting that I’ve personally seen about 6Gbps on decent hardware, with pf enabled, so I know some of this is due to to the hardware. But I also did try some troubleshooting:
- A variety of tweaks on both the host and in the VM with hardware offloading
- Using E1000 NICs. Yeah, that didn’t turn out too well
- Breaking out the VLANs on the host instead of in the VM, and passing in three individual NICs.
Nothing really made too much of a difference.
speedtest
Welcome back to the land of probably-not-very-useful-benchmarks. As before, this easily represents gigabit; my network was just busy.
cpu usage during routing
These CPU results are more within line of the CHR results above. Meaning resource-wise, pfSense takes a bit more to route at high-ish (crappy) speeds.
OPNSense
For the most part, OPNSense and pfSense performed mostly identically, to the point where it’s almost not worth the posting the results. But I will.
I will point out that I was able to get FRR going in about 20 seconds on OPNSense. Whereas on pfSense after hacking around with it for 20 minutes I still couldn’t get it running.
I’ll also say that I like some things about the OPNSense UI and absolutely hate others. There are definitely some things buried in weird and unintuitive places.
boot time
At 34 seconds with 2 seconds of bootloader, this is a few seconds quicker than pfSense.
iperf3 results
Within the same range as pfSense, similarly disappointing:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.03 sec 1.29 GBytes 1.10 Gbits/sec receiver
[ 8] 0.00-10.03 sec 1.80 GBytes 1.54 Gbits/sec receiver
[SUM] 0.00-10.03 sec 3.09 GBytes 2.65 Gbits/sec receiverspeedtest
Yet another basically-gigabit screenshot:
cpu during routing
As with pfSense, the CPU definitely spikes and stays a bit higher during heavy routing.
Final Results
After all these tests, it appears I’ll probably be sticking to VyOS or Debian for my routing needs. Mostly just for pure routing tasks, which is a lot of what I end up labbing up, Linux seems to do it a bit better.
| Router | Boot Time* | Intervlan iperf3 | WAN Speedtest** |
|---|---|---|---|
| VyOS | 37 seconds*** | 12.3 Gbps | 878Mbps down/918Mbps up |
| Debian/FRR | 19 seconds | 12.2 Gbps | 957Mbps down/932Mbps up |
| CHR | 12 seconds | 3.01 Gbps | 845Mbps down/868Mbps up |
| OPNSense | 34 seconds | 2.65 Gbps | 888Mbps down/930Mbps up |
| pfSense | 37 seconds | 2.57 Gbps | 874Mbps down/896Mbps up |
- * boot times all include bootloader waits. Linux-based was usually about 5 seconds (except CHR). FreeBSD-based is 2 seconds.
- ** All results are within a margin of error that says “gigabit-capable“. My network was somewhat busy at the time I ran the speedtests, not to mention this lab is buried behind two other different routers.
- *** I’ve seen VyOS take considerably longer once more complex configurations are applied. Upwards of a minute or more.
As mentioned, I’ve seen pfSense run up to 6-7Gbps. So the slow speeds on pfSense and OPNSense here are almost certainly due to the host. Along the same line, my VyOS routers that run on my E5-2540v4 can route at 30Gbps or faster, so the 12Gbps observed here is slow.
As far as the CHR is concerned, I’m not sure. I’ve done extensive testing with them, and even with the unlimited license, I’ve never see very good performance out of them, at least not without 100 hosts behind them.
The ESXi Mutation
So before I close this up, I decided to spin up one more set of tests on ESXi. The painfully low pfSense and CHR numbers made me really believe that there was an incompatibility somewhere. Maybe CHR and pfSense just really don’t like the virtio drivers.
I also only tested VyOS, CHR, and pfSense, since the Debian and OPNSense numbers were largely duplicative.
| Router | Boot Time | Intervlan iperf3 |
|---|---|---|
| VyOS | 32 seconds | 17.3 Gbps |
| CHR | 17 seconds | 7.63 Gbps |
| pfSense | 33 seconds | 5.61 Gbps |
As far as boot times are concerned, the shorter times are caused by being on faster storage, though I don’t know why the CHR decided to be an outlier.
For VyOS, the slightly increased number tracks with my expectations due to CPU and memory bandwidth. My ESXi server is an E5-2640v4, which runs circles around a D-1521.
For the CHR and pfSense, I wouldn’t expect the numbers to be over double just due to the platform change, so I have to believe it’s mostly due to some driver situation.
