Load balanced, redundant network configuration for Linux using ECMP, Quagga, BGP and OSPFConsider this diagram:

r1 and r2 are routers (possibly running FreeBSD & Quagga, as described here, or perhaps Cisco 7204/7206)

Each router is connected to both ISPs. Hopefully, each ISP also has each connection handled by a unique switch and router.

The two switches each service a unique network segment, possibly using private IP addresses (192.168.1.0/24 and 192.168.2.0/24)

The organisation has a block of real IP addresses which it want's external hosts to interact with. Each internal host is assigned one of the real addresses on it's loopback interface with a /32 netmask, so that no NAT is necessary.

The servers are each running Linux. The Linux kernel is compiled with Equal Cost Multi-Path routing enabled (configuration option

CONFIG_IP_ROUTE_MULTIPATH=y). This means that the kernel will permit multiple default gateways in the routing table, and will load balance outgoing traffic across them.

Routing entries, including the default gateway addresses, are not specified manually on any of the servers or routers. Instead, Quagga is running on every server. On the internal servers, Quagga uses OSPF to discover the router addresses, and then creates the default routes.

If either route becomes unavailable, because of router failure, switch failure or network card/cable failure, or during maintenance, the OSPF protocol will detect the failure within 5 seconds. When failure is detected, the route is automatically deleted from the routing table, so the kernel won't continue sending packets via that route.

Using OSPF to detect failure is ideal, because OSPF relies on packets sent at layer 3 - the IP layer. If, for instance, the cable between the 2950 and router r2 was removed, all other hosts connected to the 2950 would still feel that there was an electrical connection through the switch, and would continue sending some of their outbound packets through that switch. However, OSPF will detect this situation correctly.

Configuration details

We assume each host is running Debian Linux (except the routers). The real IP of the host is to be substituted where you see A.B.C.D in the examples.

Install Quagga on each host with the commands:

apt-get update apt-get install quagga iproute

Put the following in /etc/network/interfaces:

auto loiface lo inet loopback up ip addr add dev lo A.B.C.D/32 scope global

# notice that we use `manual' rather than `static', so that we can# over-ride the scope parameterauto eth0iface eth0 inet manual up ip link set dev eth0 up up ip addr add dev eth0 192.168.1.10/24 scope link

auto eth1iface eth1 inet manual up ip link set dev eth1 up up ip addr add dev eth1 192.168.2.10/24 scope link

Now put the following in /etc/quagga/zebra.conf:

hostname www1 password changeme enable password changeme

interface lo ip address 127.0.0.1/8 ip address A.B.C.D/32 (this is your server's real IP)

interface eth0 ip address 192.168.1.10/24 multicast

interface eth1 ip address 192.168.2.10/24 multicast

!log file /var/log/quagga/zebra.log

This is /etc/quagga/ospfd.conf

hostname www1password changemeenable password changeme

interface eth0 no ip ospf authentication-key ip ospf hello-interval 2 ip ospf dead-interval 5

interface eth1 no ip ospf authentication-key ip ospf hello-interval 2 ip ospf dead-interval 5

router ospf ospf router-id A.B.C.D network 192.168.1.0/24 area 0 network 192.168.2.0/24 area 0

!log file /var/log/quagga/ospfd.log

Modify /etc/quagga/daemons.conf: set zebra=yes and ospfd=yes

Testing

Once configured, reboot your host.

Type ip route and you should see a list of routes showing multiple default gateways.

Try unplugging one of the routers - then check the routing table on one of the servers. After 5 seconds, the references to the unplugged router should be gone from the routing tables.

Troubleshooting

Make sure Equal Cost Multi-path is enabled in each Linux kernel Make sure Multicast is enabled in the kernel. Check that the network cards and

drivers support multicast - some don't Make sure that you don't have iptables blocking the OSPF packets. OSPF sends

packets using IP protocol 89 (not TCP or UDP). This allows OSPF:

iptables --insert INPUT -s 192.168.0.0/16 --protocol ospf -j ACCEPT

http://www.readytechnology.co.uk/open/bgp/loadbalanced.html