SIMPLE LIFE, COMPLICATED MIND

Multiple default routes gateways with two different ISP ipfw PF setfib load balancing

Thanks to phoenix help I was able to setup multiple default routes, or a default route per network/interface to be precise, in Debian/Linux it is as simple as that:

# cat /etc/network/interfaces

iface eth0 inet static    address 10.0.0.2    netmask 255.255.255.252    gateway 10.0.0.1iface eth1 inet static    address 20.0.0.2    netmask 255.255.255.252    gateway 20.0.0.1

That would be example topology (but more than 2 interfaces is also possible).

ISP NETWORK 0                   ISP NETWORK 1         \                          /          \                        /           \                      /          ROUTER 0            ROUTER 1          10.0.0.1/30         20.0.0.1/30              \                  /        +------\----------------/------+        |       \              /       |        |       wan0          wan1     |        |    10.0.0.2     20.0.0.2     |        |                              |        |   FREEBSD (Firewall Router)  |        |                              |        |             dmz0             |        |       192.168.0.1/24         |        |       192.168.0.2/24         |        +------------------------------+                       |                   +--------+                   | switch +                   +--------+                       |                      /               ______/              /    +--------/-----+    |       /      |    |      lan0    |    |  192.168.0.3 |    |  192.168.0.4 |    |              |    | Web Server 0 |    +--------------+

Now, You can not use the ‘casual‘ defaultrouter="X" cause it will be only for one network.

We will have to use setfib(1) to create two (or more) separete routing tables per network/interface.

Note: FIB (Forward Information Base, synonym for a routing table here)

Add these lines to /boot/loader.conf file:

# vim /boot/loader.conf
### select one of following firewalls to enable.
#ipfw_load="YES" # Firewall
pf_load="YES" # packet filter
pflog_load="YES" # packet filter log

### set number of routing tables to support.
net.fibs=2

Note: to view a list of options: cat /boot/defaults/loader.conf.

It will unfortunately require kernel recompile, but its not as that hard:

# cd /usr/src/sys/`uname -m`/conf
# cp GENERIC MYKERNEL8.2

# vim MYKERNEL8.2
options ROUTETABLES=2 # max 16. 1 is back compatible.

Note: to view a list of available options:
# cat /usr/src/sys/conf/NOTES | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.

# cat /usr/src/sys/`uname -m`/conf/DEFAULTS

# cd /usr/src
# make buildkernel KERNCONF=MYKERNEL8.2
# make installkernel KERNCONF=MYKERNEL8.2

# sync ; reboot

Note: if the system could not boot properly, press any keys other than enter key when you see the count down number. and type following at boot: prompt:
boot: unload
boot: /kernel.old
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/boot-blocks.html

Remove the files generated during recompiling the kernel. 
# rm -rf /usr/obj/usr/src/sys/MYKERNEL8.2

After the kernel has been built and rebooted the different routing tables can be accessed as shown in the setfib(1) man page by issuing command setfib 0 netstat -rn. 0 is the default routing table.

After this one has to create the second routing table by prepending every route add command with setfib 1 route add… e.g:

# setfib 0 /sbin/route add -net default 10.0.0.1
# setfib 1 /sbin/route add -net default 20.0.0.1

Note: with packet filter one can control how the routing table is selected by using rtable option but it should be noted that this selection can only be done on the input of the packets as the routing decision is done at the input not at the output.

# vi /etc/rc.conf
### WAN interfaces. You do not need to set defaultrouter="XXX" here.
ifconfig_wan0="inet 10.0.0.2/32"
ifconfig_wan1="inet 20.0.0.2/32"

### Note: commented out for multiple WAN routes setup. Routes will be added in /etc/rc.local file.
#defaultrouter="10.0.0.1"

### LAN static route
#static_routes="lan"
#route_lan="-net 192.168.50.0/24 192.168.0.11"

### PF (Packet Filter Firewall)
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"
pflog_flags=""

### Enable as LAN gateway
gateway_enable="YES"

All the rest configuration resides in /etc/rc.local file:

# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 10.0.0.1

### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 20.0.0.1

### assing route tables to interfaces
### Note: uncomment following lines if you are using ipfw. Leave it if you are using PF.
###
#ipfw -f flush
#ipfw add allow ip from any to any via lo0
#ipfw add setfib 0 ip from any to any via wan0
#ipfw add setfib 1 ip from any to any via wan1
#ipfw add allow ip from any to any

These would be handy for restarting:
# /etc/rc.d/netif restart && /etc/rc.d/routing restart
# /etc/rc.d/local restart
# pfctl -f /etc/pf.conf

View the fib:
# sysctl -a | grep fib
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 2

View the default route of each routing table:

# setfib 0 netstat -rn
# setfib 1 netstat -rn

Try to ping:
# setfib 0 ping google.com
# setfib 1 ping google.com

===============================================================
pf.conf

Looking for something like this?

# man 1 setfib
# man 2 setfib

And pf.conf(5) has the route-to and reply-to directives, of course.

# man 5 pf.conf | less +/rtable
# man 5 pf.conf | less +/route-to
# man 5 pf.conf | less +/reply-to

# vim /etc/pf.conf
### [VARIABLES]
wan_if0 = "wan0"
wan_if1 = "wan1"
dmz_if0 = "dmz0"
www0_ip0 = "192.168.0.3"
www0_ip1 = "192.168.0.4"

##########################
##### TABLES - A structure used to hold lists of IP addresses.
##########################
table <blocked_ip> persist file "/etc/pf-blocked_ip"

### [NAT]
nat on $wan_if0 from $dmz_if0/24 to any -> $wan_if0
nat on $wan_if1 from $dmz_if0/24 to any -> $wan_if1

### [HTTP] RDR Outside to DMZ
rdr on $wan_if0 proto tcp from any to $wan_if0 port 80 -> $www0_ip0
rdr on $wan_if1 proto tcp from any to $wan_if1 port 80 -> $www0_ip1

### [HTTP] outside to router
pass in quick on $wan_if0 proto {tcp} from any to $wan_if0 port 80 rtable 0
pass in quick on $wan_if1 proto {tcp} from any to $wan_if1 port 80 rtable 1

### [HTTP] router to dmz
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip0 port 80 rtable 0
pass out quick log on $dmz_if0 proto {tcp} from any to $www0_ip1 port 80 rtable 1

Troubleshooting
dump traffic on a network, see packets that match a certain port
# tcpdump -ni re0 port 53

You may also want to limit tcpdump to just show icmp:
# tcpdump -ni re0 icmp

Or just to/from a certain host:
# tcpdump -ni re0 icmp host 172.16.70.12

Reading a PF (packet filter) log file

The log file written by pflogd is in binary format and cannot be read using a text editor. Tcpdump must be used to view the log.

To view the log file:

# tcpdump -n -e -ttt -r /var/log/pf.log

Note: that using tcpdump(8) to watch the pflog file does not give a real time display. A real time display of logged packets is achieved by using the pflog0 interface:

# ifconfig | grep pflog
# tcpdump -n -e -ttt -i pflog0

Filtering Log Output

Because pflogd logs in tcpdump binary format, the full range of tcpdump features can be used when reviewing the logs. For example, to only see packets that match a certain port:

# tcpdump -n -e -ttt -r /var/log/pf.log port 80

set up PF (packet filter) firewall on FreeBSD 8.1

Other Solution:
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-aggregation.html
http://www.freebsd.org/cgi/man.cgi?query=lagg&sektion=4
http://scratching.psybermonkey.net/2009/06/freebsd-combine-2-or-more-nic-using.html
===============================================================
Just a quick note, if doing this on a router with a single internal interface.

Traffic originating on the local network will go out the default route of FIB 0.

Only traffic coming in on the second public interface will go out the same interface.

IOW, the setup above is really only useful for incoming traffic, to make sure that it goes back out the correct interface.

However, a few more IPFW rules can be added to classify traffic on the internal NIC.

It all depends on what you want to accomplish.

Yes, its mainly for that purpose, to assign proper gateways to networks/interfaces (which is a lot more easier @ Debian in /etc/network/interfaces)

We may even use FIB# 1 and 2 for these networks, and use FIB# 0 with other NIC as a default interface, posibilities are of course big, but I wanted to point posibility of having a gateway per network.

http://www.daemonforums.org/showthread.php?t=4610
===============================================================
setfib selects the routing table for locally originated
outgoing packets. Besides locally originated packets, there
are packets arriving from the network and need to be forwarded.
These packets can be classified in a specific routing table
with the aid of ipfw. That‘s all there is. I can‘t think
of something else that needs to be thought with regard to
multiple routing tables.

HTH, Nikos

http://lists.freebsd.org/pipermail/freebsd-questions/2009-July/202462.html
===============================================================
Something has been bugging me for several years now. In that time I have usually had access to multiple WAN connections, owing to my participation in the telecom industry. However, I‘ve never been able to get SSHD to behave the way I wanted it to. I wanted to be able to connect to the SSH daemon on my (FreBSD) router from whichever WAN connection I wanted. Unfortunately, SSHD is stuborn about always routing its response to the default gateway of the router, which breaks an SSH connection coming in from the secondary WAN connection.

I have finally, at long last, found the solution.

The Problem: When describing this problem to other FreeBSD users, they pretty universally assumed that I was mistaken, and that the SSH daemon would by default route its responses out the interface that the initial request had been received on. Evidently, it is uncommon to run SSHD on more than one WAN interface. At its root, this problem just boils down to the routing table. SSHD doesn‘t route it‘s responses to the interface that they were received on, it uses the routing table to determine where to send it‘s responses. If the request came from a local network, then it responds to the local network. If it originated from a non-local network, then it uses the default gateway. It‘s really that simple. There is no logic for having multiple paths to non-local networks.

Frequently Offered Solution: Since I use pf for my firewall, I‘m frequently told to use pf‘s route-to and reply-to functionality to solve this problem. I have at times used route-to and reply-to extensively in my pf.conf. But route-to and reply-to do not trump the default routing table for traffic the originates or terminates on the router itself. They are useful only for traffic passing through the router. pf can only make routing decisions when a packet passes through an interface. It can try and set the reply-to interface to be the second WAN connection when an inbound SSH connection is made, but neither the SSH daemon nor the routing table on the host know or care about the routing preferences of pf.

The Real Solution: FreeBSD has support for multiple routing tables. It‘s little known, and even less documented, but it does exist. Basically, you need to recompile your kernel with multiple routing table support ("options ROUTETABLES=2"), and then use the setfib program to set which routing table to use when starting another program. The syntax is similar to nice: setfib 1 route add default 192.168.1.1 would add a default route of 192.168.1.1 to the second routing table on the host. If not specified, the default routing table is 0. On FreeBSD, pf also has support for multiple routing tables with the little discussed rtable option.

So here are the steps to solving this problem:

Step 1: Rebuild your kernel with the ROUTETABLES option set to a non-zero integer. This is how many routing tables your host will support.

# cat /usr/src/sys/`uname -m`/conf/MYKERNEL8.2 | grep ROUTETABLES
options ROUTETABLES=2 # max 16. 1 is back compatible.

Step 2: Disable the SSH daemon in your rc.conf:

# cat /etc/rc.conf | grep ssh
#sshd_enable="YES" # This is now handled by /etc/rc.local

Step 3: Create /etc/rc.local file to start multiple SSH daemons. To do this, copy the /etc/ssh/sshd_config file to several alternates, one per interface you want SSHD to listen to, and set the ListenAddress for each file to only the IP for that interface.

# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable0
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.rtable1

# vim /etc/ssh/sshd_config.rtable0
ListenAddress 20.0.0.2
ListenAddress 192.168.0.1

# vim /etc/ssh/sshd_config.rtable1
ListenAddress 30.0.0.2
ListenAddress 192.168.0.2

Note: Multiple ListenAddress options are permitted.

# vim /etc/rc.local
### add first routing table for first interface for first ISP network.
/usr/sbin/setfib 0 /sbin/route delete default
/usr/sbin/setfib 0 /sbin/route add default 20.0.0.1

### add second routing table for second interface for second ISP network.
/usr/sbin/setfib 1 /sbin/route delete default
/usr/sbin/setfib 1 /sbin/route add default 30.0.0.1

### Start SSH daemons for each interface
/usr/sbin/setfib 0 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable0
/usr/sbin/setfib 1 /usr/sbin/sshd -f /etc/ssh/sshd_config.rtable1

Step 4: Add rtable awareness to your pf.conf file:

[root@router]~ # cat /etc/pf.conf | grep rtable
pass in log on tun0 inet proto icmp from any to (tun0) icmp-type rtable 0
pass in log on tun1 inet proto icmp from any to (tun1) icmp-type rtable 1
pass in log on tun0 inet proto tcp from any to (tun0) port ssh rtable 0
pass in log on tun1 inet proto tcp from any to (tun1) port ssh rtable 1
pass in log on wan0 inet proto tcp from wan0:network to (wan0) port 22 rtable 0

Conclusion: Because the SSH daemon listening on tun1 is using a routing table that features the tun1 interface as the default gateway, the response will go out tun1. An inbound connection to tun0 will hit the SSH daemon listening on tun0 (which is an entirely separate process from the one listening on tun1) and uses the routing table associated with tun0, which features tun0 as the default gateway.

In my above config it‘s worth pointing out that it doesn‘t actually matter which routing table the SSH daemon listening on the LAN interface uses, because both routing tables see the LAN network as a local one. By default on FreeBSD with multiple routing tables enabled, all local networks will still appear in all the routing tables. There is a sysctl option to disable this behavior.

High Availability HA cluster solution for FreeBSD

HAST + CARP + ZFS

HAST (Highly Available Storage) - allows to transparently store data on two physically separated machines connected over the TCP/IP network.

CARP (Common Address Redundancy Protocol) - allows multiple hosts to share the same IP address. In some configurations, this may be used for availability or load balancing. Hosts may use separate IP addresses as well, as in the example provided here.

# man carp
# man ng_one2many
# man lagg - link aggregation and link failover interface.
# man ngctl - netgraph control utility.

http://www.freebsd.org/doc/handbook/carp.html
http://forums.freebsd.org/showthread.php?t=17133
http://blather.michaelwlucas.com/archives/241
===============================================================

Reference:
http://gala4th.blogspot.com/2011/10/multiple-default-routes-gateways-with.html
High Availability HA cluster solution for freebsd
http://www.daemonforums.org/showthread.php?t=4610
http://freebsd-forums.liquidneon.com/showthread.php?t=20166
http://www.mmacleod.ca/blog/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/
http://www.melen.org/u/jan/wp/?p=102
http://gala4th.blogspot.com/2010/12/set-up-pf-packet-filter-on-freebsd.html
Running PF & IPFW Together
Prevent default gateway route and resolv.conf being overwritten by DHCP