OpenBSD part VI: CARP.

Let’s start with a note to self: when copying virtual machines, be sure to generate the MAC addresses again, otherwise you may end up with two virtual machines sharing the same MAC address. That explains why CARP wouldn’t run at the first try.

But what is CARP? It stands for Common Address Redundancy Protocol, and works the same as HRSP, VRRP and GLBP: it allows several routers to share a virtual IP which acts as the gateway for connected hosts. When one of the routers fails, another takes over the virtual IP so network connectivity for the hosts remains.

CARP has quite a history, which you can read in detail on Wikipedia. Because of it, CARP uses the same IP protocol as VRRP (112) and thus will show up as VRRP in Wireshark.

Configuration, with persistence between reboots, is similar to the interface configuration and bridging setup: CARP uses a special interface which, you guessed it, is created at boot if the file /etc/hostname.carp0 is found. However, I was unable to find the correct syntax in the file for OpenBSD 5.0, and the ones suggested in the manuals don’t work. However, just having the file already creates the interface, and everything with an exclamation mark in front of it inside a hostname file will be executed as a command, so the following line works:

• !ifconfig carp0 vhid number ip-address netmask subnetmask
• The vhid number is the CARP group. You can have more than one CARP group per interface, but for a given group, the configuration has to be the same on all devices.
• ip-address is the virtual IP that can be used as gateway for the hosts on the subnet.
• Optionally, you can add ‘pass password‘ in the command to secure the CARP packets with a password.
• Also optional, ‘advskew number‘ is a value between 0 and 254. The OpenBSD CARP with the lowest advskew value becomes the master.

Other options are possible, but these are the most important to get everything going. If things don’t work yet, it’s likely that pf is blocking the CARP packets. ‘pass in quick on em0 proto carp’ and ‘pass out quick on em0 proto carp’ solve this. Keep in mind all filtering still has to be done on the physical interface, filtering anything on ‘carp0’ will not take effect.

Finally, just like with the other gateway redundancy protocols, there’s a preempt option. When preempt is disabled, the first active OpenBSD will become master, even if other OpenBSD’s with a lower advskew value become active. When it’s enabled, the OpenBSD with the lowest advskew value will become master, whether the currently active OpenBSD has failed or not. The value can be manipulated in /etc/sysctl.conf, where net.inet.carp.preempt has to be set to ‘1’ (or just remove the ‘#’ if it’s already present but commented out).

Since I’ve covered enough for a complete setup, my next post will not be about OpenBSD anymore. Stay tuned!

Advertisement

Using IP SLA and tracking to influence the gateway.

In an earlier post I already described how to set up IP SLA check a connection and how to bind it to a tracking object. Since then I’ve found out some differences between the IOS used on switches and the one used on routers. I used a hidden command ‘track 2 rtr 20’ to make a tracking object out of the IP SLA, but on a router, this command is not hidden. Also, on a layer 3 switch the IP SLA commands all start with ‘ip sla’, whereas on a router it’s ‘ip sla monitor‘. It may also be that my router is an older (EOL) device while my switch is relatively new. I guess I’ll have to know both to be sure.

I was also asked some interesting questions by Tomasz, and decided to expand the topic in a new post, describing some more options for these tracking objects. He also asked if it could be used as legal proof in case of a dispute. I’m not sure of it in case of my own ISP, but I think that on a corporate level, if enough details are mentioned in the contract, this may indeed be the case. After all, it’s an objective measurement of the connection.

So besides the simple logging, other things are possible. Take for example the following configuration:

The two routers have HRSP configured for the user subnet. Both routers have preempt configured. The left router has priority 200, the left router 190. The result is that in a normal situation, the left router will be the active router.

You can bind a tracking object to HRSP, for exampe, track the state of a connected interface. You can set that if an interface is down, priority of the router is decreased by 20. Okay, but is that really a safe solution? If both routers have a different number of connected interfaces, how do you determine which ones are important? And, say those interfaces stay up, but the problem occurs one hop away? It will go undetected, the link will stay up, packets will be sent through it, and dropped at the next hop.

Of course, you can also track the presence of certain routes in the routing table. This makes a lot more sense already: if the subnet can be reached, everything is fine, if it can’t, something is wrong, and it’s better that the other router takes the active role in that case. But it might still be insufficient for some types of data. There will be convergence by the routing protocols, which takes time. A network redesign or failure might change the route advertised into a more or less specific route (bigger or smaller subnet mask). E.g. a route to 10.0.1.0/24 is present in the routing table pointing to f0/1, the route is being tracked. Something fails in the network, the route is removed from the routing table, but a working default route pointing to f0/2 is still present. Despite connectivity the router will still lose HRSP active status! And what if the route still works but suddenly experiences very high latency, while the other HRSP router is working fine?

IP SLA is a perfect solution here. The only downside is that it takes a bit of CPU on the router and some bandwidth, but we’re talking about just a few packets. For example, we configure the IP SLA to track the state of an important production server (10.0.5.5) every 3 seconds using pings. If high latency (+100ms) is experienced or the ping fails, HRSP priority will decrease by 20.

Router(config)#ip sla monitor 50
Router(config-sla-monitor)#type echo protocol ipIcmpEcho 10.0.5.5
Router(config-sla-monitor-echo)#frequency 3
Router(config-sla-monitor-echo)#timeout 100
Router(config-sla-monitor-echo)#exit
Router(config)#track 5 rtr 50
Router(config-track)#exit
Router(config)#interface FastEthernet0/0
Router(config-if)#standby 1 preempt
Router(config-if)#standby 1 ip 192.168.0.1
Router(config-if)#standby 1 priority 200
Router(config-if)#standby 1 track 5 decrement 20
Router(config-if)#end

Note that you can track more than one object. So you can do this for multiple important points in the network. The router that can reach the most of these points with the least latency will have the active HRSP role. This ensures high availability of network services to users. Don’t forget the preempt command, otherwise the routers will not change active roles until one fails completely.

VRRP between Cisco and Vyatta.

I already mentioned in an earlier post that I was doing some experiments with Virtual Router Redundancy Protocol on routers from different vendors. For those of you not familiar with VRRP: it’s a protocol that allows multiple routers to share the same IP address, which then can be used as the default gateway for end devices. This gives you some redundancy in case a router goes down. VRRP is the IETF standard for an earlier protocol: HRSP, which is available on Cisco devices only. For more info, Wikipedia is your friend.

I already managed to get VRRP running on both GNS3 and real Cisco routers, but since it’s supposed to be a standard, why not try it in a multivendor environment? My favorite non-Cisco router is Vyatta: it’s a stripped-down Linux with nothing left but the kernel and network-related packages already installed. The command line handles somewhat like a Cisco IOS. Since it can run on almost any x86 hardware, you can virtualize it too, so it’s an easy solution in my lab. The basic version is free.

I followed the guide I found on openmaniak.com and got it running in no time. I used the following configuration: 192.168.0.2 for the Vyatta, 192.168.0.3 for the Cisco router (a 2611 running 12.3 IOS), and 192.168.0.5 as the virtual IP address.

I started and configured my Vyatta first. Here you see it sending VRRP multicasts to 224.0.0.18, to announce to the other router(s) that he’s currently the master and will handle all packets send to 192.168.0.5, the virtual address.

Next, I booted and configured the Cisco router. Note that both configurations used the ‘preempt’ command, which means that if a ‘better’ router is present in the subnet, it will immediately assume the master role, instead of waiting until the current leader (the Vyatta) goes down. The ‘better’ router here means the one with the higher priority, or in case of a tie, the one with the highest IP address.

Since the Cisco router has a higher IP address, it takes the master role after a few seconds:

After the Cisco router becomes the master, it will handle any packet destined for 192.168.0.5. Should the router fail, for example by me unplugging the ethernet cable, the Vyatta router will take the master role again and the address 192.168.0.5 will stay reachable.

So far so good. But I did come across a small problem in my tests. if you watch both images closely, you’ll notice that the Cisco router is using a source MAC address of 00:00:5e:00:01:01. This is correct, because this is the MAC address that must be used according to the RFC. (The last ’01’ is the VRRP group 1. Had I configured it with VRRP group 5, it would be ’05’.)

The Vyatta router does not use this MAC address but instead uses his own (00:0c:29:fd:d5:23, VMWare virtual NIC). I’ve done some research around the web and could not find anything conclusive, but I’ve heard of Linux versions having trouble using multiple source MAC addresses, so this may be the cause. It does create a problem though, because if a router fails in this configuration, the end devices are left with the wrong ARP information, making the 192.168.0.5 address unreachable after all. It’s possible to solve this issue by sending a Gratuitous ARP packet in case of a failure, but I didn’t notice such a packet in my tests, and it would still make things more complex than they are supposed to be. At this moment I am uncertain if VRRP works well in Vyatta. But that aside, I did learn a lot today.