In this series of posts, we’ll be working through building a routing lab-within-a-lab featuring VyOS. If you are unfamiliar with VyOS, you can read more about it here.

Over the last few months, I’ve switched all the routing for my homelab to virtual VyOS instances. The main reason was to really get my hands dirty with some more advanced routing concepts.

As I have multiple hypervisors, this configuration creates a different routing VM on each hypervisor for redundancy and backup. This setup allows me to do a number of things:

Wire-speed routing at 10Gb without noisy and power-hungry switches. Now I just use cheap basic managed switches (specifically the CRS317).
30Gbps or higher routing between VMs on the same host.
Ultimate redundancy and backup for both IPv4 and IPv6.

First Goal

Our first goal will be to configure a simple router+LAN setup that would mimic most home designs. For now, we’ll be skipping the firewall and just worrying about routing. The pieces:

Edge routing VM. This is the edge device of our little routing lab. A simple VyOS instance with two interfaces. One on your existing network to act as the “WAN”, and a second interface to be the routing “LAN”. Since it’s virtualized, you could also mimic this with a single interface and a “trunk” port.
A simple Debian/CentOS/etc VM to act as a client on our LAN. It makes it easy to run traceroutes, MTRs, speed tests, and whatever other routing validation we can imagine. You could even move an existing Windows VM or Linux install just by changing the port group the VM uses.

There are a few requirements before we get started:

A hypervisor where you can easily set up “virtual switches”. I’m using ESXi.
A Debian/CentOS/LiveCD VM to be our test “client”. I install a GUI so I can use Chrome to run speedtests or verify Internet connectivity. Again, any existing VM can temporarily be used for this with a simple port group/network migration.
The VyOS ISO. At the time of this writing, I prefer the latest RC7 as it has FRR for the routing suite.
Maybe 5-10GB of RAM and 5-10GB of disk space.
The ability to create a static route on your main firewall/router. We could do NAT, but the goal here is pure routing.
A /24 subnet (or smaller/bigger), that’s not in use otherwise in your network. I will be using 10.222.222.0/24

Let’s Go!

The first thing you will want to do is create a new virtual switch to use as our new LAN. It doesn’t need any uplinks or external connectivity. With ESXi, you would first create a new vSwitch and remove the uplink from it.

And then create a new Port group and select the vSwitch you just created (with vCenter this can be done in one step).

Once that’s done, we create our Edge VM. This will be a VyOS instance where we will run things like our DHCP server for the new LAN. In a real setup, this VM or physical box would be connected to your ISP, do firewalling/NAT, etc:

A few of the important things follow. VyOS doesn’t need much memory or disk, and at least for RAM, 1GB is probably overkill. 2GB of disk is the new minimum for VyOS. The ISO that I mounted is vyos-1.2.0-rc7-amd64.iso, but I’d grab the latest version that’s available.

For networking, my “Network Adaptor #1” is hooked to my existing LAN. The new network adaptor is connected to the new LAN that is going to be used for testing. These should both be VMXNET3 for best performance.

It’s important to note that everything with VyOS is done at the CLI (that’s generally fairly intuitive). There is no GUI to access, so initially you’ll have to access it entirely through your hypervisor console. SSH access can be configured once you have networking set up.

The first thing to check before you start up the VM is whether ESXi changed around your network interfaces for you. It did for me:

I want my “WAN” to be the first network interface in the VM, and the RouteLAB-LAN to be the second one. Easy enough to switch them back:

Edge VM

With the edge VM created, it’s time to get VyOS installed. Boot it up and log in at the console with vyos / vyos as the username and password. Entering install image will begin the installation.

You should be able to accept the defaults:

Name the image, and choose a password. VyOS uses squashfs images, so if an install goes badly, usually you can roll back to the last working copy:

And after grub is installed, you should reboot to boot into your new system:

Edge VM part 2

Before you can go any further, you need to decide on a static IP for your new VM. This will be an IP on your existing network. If we were doing NAT or dynamic routing, you could probably just use your existing DHCP, but at least for now, we want static routing.

My config looks like:

Existing LAN IP Range: 10.0.1.0/24
VyOS Edge IP: 10.0.1.222
LAN Gateway (my existing firewall): 10.0.1.1
DNS servers: 8.8.8.8, 1.1.1.1

For the RouteLAB-LAN:

VyOS Edge IP: 10.222.222.1/24, this will be the gateway for the hosts on the RouteLAB-LAN
DHCP Server running the range of 10.222.222.100-200
DNS servers: 8.8.8.8, 1.1.1.1

You also have to tell your existing network how it will reach your new RouteLAB-LAN

This also means on your existing firewall/router, you need a new static route. The route will be the destination network of 10.222.222.0/24 with the gateway/next-hop of 10.0.1.222. How to do this is dependent on your router/firewall. With something like pfSense, you might use these instructions.

Edge VM config

The config we are setting up is VERY simple and involves no firewalling, which obviously would be a requirement in real usage. But it is just enough to get our VM online.

configure
set interfaces ethernet eth0 address '10.0.1.222/24'
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth1 address '10.222.222.1/24'
set interfaces ethernet eth1 description 'RouteLab-LAN'
set protocols static route 0.0.0.0/0 next-hop 10.0.1.1
set service ssh port '22'
set system name-server 8.8.8.8
set system name-server 1.1.1.1
commit
save
exit

Hopefully it should be fairly self-explanatory:

Enter configure mode.
Set the IP on your existing network and a description for the interface. This is the IP on your existing LAN for this device. This is eth0.
Set the IP on our RouteLAB-LAN and the description for the interface. This is eth1.
Set a static route for 0.0.0.0/0. This is like setting the gateway address; it tells your VM how to get to the Internet and the rest of your network.
Turn on SSH. You will then be able to access this VM over SSH with the username vyos and the password you set up earlier. There are settings available to use a different port, username, rate-limit, or even key-based authentication.
Sets the DNS servers for the VM. Not strictly needed, but nice if you want Internet access on this VM.
commit activates the config.
save saves the config so it will persist through a reboot.
Exit configure mode.

At this point you should be able to ping/ssh the existing LAN address from anywhere on your network. Pinging the 10.222.222.1 address successfully will also tell you that the static route is working. If something isn’t working, there are a few possibilities:

Your static route isn’t setup correctly. In my above config, this would mean I am able to ping 10.0.1.222 but not 10.222.222.1 from other hosts on my LAN
Your firewall is restricting something
If pings to both addresses work, but the Edge VM doesn’t have access to the Internet in general (like pinging 8.8.8.8), it’s probably because you need to adjust your outbound NAT. In pfSense you’d want to switch from Automatic to Hybrid or Manual Outbound NAT and allow 10.222.222.0/24. We technically aren’t going to need Internet access for this project, but it’s nice when your Client VM is able to access the wide-Internet.

Edge VM config part 2

The final piece to this part of the project is getting DHCP working for our RouteLAB-LAN.

configure
set service dhcp-server shared-network-name RouteLAB-LAN authoritative
set service dhcp-server shared-network-name RouteLAB-LAN subnet 10.222.222.0/24 default-router 10.222.222.1
set service dhcp-server shared-network-name RouteLAB-LAN subnet 10.222.222.0/24 dns-server 8.8.8.8
set service dhcp-server shared-network-name RouteLAB-LAN subnet 10.222.222.0/24 dns-server 1.1.1.1
set service dhcp-server shared-network-name RouteLAB-LAN subnet 10.222.222.0/24 lease 86400
set service dhcp-server shared-network-name RouteLAB-LAN subnet 10.222.222.0/24 range 0 start 10.222.222.100
set service dhcp-server shared-network-name RouteLAB-LAN subnet 10.222.222.0/24 range 0 stop 10.222.222.200
commit
save
exit

Again, this config should be fairly self-explanatory:

Enter configure mode.
Set up this DHCP server as authoritative.
Name this VM’s 10.222.222.1 IP as the default gateway for the subnet.
Set up a few different name-servers.
Set our lease length to a day.
Set our range of addresses from clients will grab addresses.
commit and save.
Exit configure mode.

Now if you change the port group on your Client VM (and possible disable and re-enable the network interface from within the VM):

You should see your client VM pop up in the DHCP leases on the Edge VM with a show dhcp server leases:

[email protected]:~$ show dhcp server leases
IP address      Hardware address    Lease expiration     Pool          Client Name
--------------  ------------------  -------------------  ------------  -------------
10.222.222.100  00:50:56:9d:74:42   2018/11/15 16:28:08  RouteLAB-LAN  debian

And in my case, I’m able to ping the 10.222.222.100 address from the rest of my network, and vise-versa.

End of Part 1

If you’ve made it this far, you now have a basic “lab within a lab” running. What this demonstrates is how to use a static route to avoid using the dreaded double-NAT. If you set up some firewall rules, and you plug an interface into your ISP modem or device, you could use this VM as the router for your entire network.

In part 2, we’ll work on setting up some redundancy and dynamic routing within our tiny little lab.