FreeBSD - multiple routing tables

To enable multiple routing tables in FreeBSD, kernel needs recompiled with the number of routing table it should host. Following kernel option needs to be added to kernel file:

Code:

options ROUTETABLES=4

So 4 routing tables in my case. To verify number of routing tables supported by FreeBSD kernel, use sysctl:

Code:

# sysctl net.fibs
net.fibs: 4

Let's look at other FIB (should be called read) related sysctls:

Code:

net.my_fibnum: 0
net.add_addr_allfibs: 0
net.fibs: 4

From FreeBSD manual:

Quote:

Add the following kernel configuration option and rebuild the kernel. The 2 is the number of FIB (Forward Information Base, synonym for a routing table here). The maximum value is 16.

options ROUTETABLES=2

The procedure for rebuilding the FreeBSD kernel is described in the FreeBSD Handbook.

This number can be modified on boot time. To do so, add the following to /boot/loader.conf and reboot the system:

net.fibs=6

Set a loader tunable net.my_fibnum if needed. This means the default number of routing tables. If not specified, 0 will be used.

Set a loader tunable net.add_addr_allfibs if needed. This enables to add routes to all FIBs for new interfaces by default. When this is set to 0, it will only allocate routes on interface changes for the FIB of the caller when adding a new set of addresses to an interface. Note that this tunable is set to 1 by default.

The most interesting one is "add_addr_allfibs" which controls whether or not to add direct routes for an interface to all FIBs (all routing tables). Example of behavior:

Code:

# sysctl net.add_addr_allfibs=1
net.add_addr_allfibs: 0 -> 1
# setfib 1 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
# setfib 2 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire

FIB 1 has no routes. If I create a vlan interface and assign it an IP address (without selecting the FIB number), it will be added to all FIBs:

Code:

# ifconfig vlan201 create
# ifconfig vlan201 vlandev em0 vlan 201
# ifconfig vlan201 up
# ifconfig vlan201 10.1.201.2/24
# setfib 2 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
10.1.201.0/24      link#13            UC          0        0 vlan20
# setfib 1 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
10.1.201.0/24      link#13            UC          0        0 vlan20

What I did above was to create a vlan201 tagged vlan interface, bring it up and assign an IP address to it, all in the default routing table. Having the net.add_addr_allfibs sysctl enabled, it allowed me to add direct routes (/24) to all routing tables.

Same test, but with the sysctl disabled:

Code:

# sysctl net.add_addr_allfibs=0
net.add_addr_allfibs: 1 -> 0
# ifconfig vlan201 destroy
# ifconfig vlan201 create
# ifconfig vlan201 vlandev em0 vlan 201
# ifconfig vlan201 up
# ifconfig vlan201 10.1.201.2/24
# setfib 2 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
# setfib 1 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Net

And the direct /24 routes are not in FIB1 and FIB2.

What is this good for ? Well it helps when you want to send traffic which destined for em0 via em1 interface. Using a single routing table, or direct routes in all FIBs, will make the FreeBSD kernel use the loopback interface to send packets sourced with IP of em1 to IP of em0, but with multiple routing tables and the sysctl disabled, you can easily send via one interface (it can be vlan interface or physical interface) through the external network and have it returned on another interface, sending the reply through the last interface (ping request via em0, reply via em1, or syn via em0, syn+ack via em1).

Note, for TCP/UDP, support for FIB needs to be used (nginx uses "setfib" configuration stanza) or daemon needs to be started under that FIB number.

Linux, FreeBSD, Juniper, Cisco / Network security articles and troubleshooting guides

Internet Protocol, Transport Control Protocol, Network protocols, Routing, Routers, IP aliases, Routes, Ethernet

FreeBSD - multiple routing tables

FreeBSD - multiple routing tables