Let’s start with a note to self: when copying virtual machines, be sure to generate the MAC addresses again, otherwise you may end up with two virtual machines sharing the same MAC address. That explains why CARP wouldn’t run at the first try.
But what is CARP? It stands for Common Address Redundancy Protocol, and works the same as HRSP, VRRP and GLBP: it allows several routers to share a virtual IP which acts as the gateway for connected hosts. When one of the routers fails, another takes over the virtual IP so network connectivity for the hosts remains.
CARP has quite a history, which you can read in detail on Wikipedia. Because of it, CARP uses the same IP protocol as VRRP (112) and thus will show up as VRRP in Wireshark.
Configuration, with persistence between reboots, is similar to the interface configuration and bridging setup: CARP uses a special interface which, you guessed it, is created at boot if the file /etc/hostname.carp0 is found. However, I was unable to find the correct syntax in the file for OpenBSD 5.0, and the ones suggested in the manuals don’t work. However, just having the file already creates the interface, and everything with an exclamation mark in front of it inside a hostname file will be executed as a command, so the following line works:
- !ifconfig carp0 vhid number ip-address netmask subnetmask
- The vhid number is the CARP group. You can have more than one CARP group per interface, but for a given group, the configuration has to be the same on all devices.
- ip-address is the virtual IP that can be used as gateway for the hosts on the subnet.
- Optionally, you can add ‘pass password‘ in the command to secure the CARP packets with a password.
- Also optional, ‘advskew number‘ is a value between 0 and 254. The OpenBSD CARP with the lowest advskew value becomes the master.
Other options are possible, but these are the most important to get everything going. If things don’t work yet, it’s likely that pf is blocking the CARP packets. ‘pass in quick on em0 proto carp’ and ‘pass out quick on em0 proto carp’ solve this. Keep in mind all filtering still has to be done on the physical interface, filtering anything on ‘carp0’ will not take effect.
Finally, just like with the other gateway redundancy protocols, there’s a preempt option. When preempt is disabled, the first active OpenBSD will become master, even if other OpenBSD’s with a lower advskew value become active. When it’s enabled, the OpenBSD with the lowest advskew value will become master, whether the currently active OpenBSD has failed or not. The value can be manipulated in /etc/sysctl.conf, where net.inet.carp.preempt has to be set to ‘1’ (or just remove the ‘#’ if it’s already present but commented out).
Since I’ve covered enough for a complete setup, my next post will not be about OpenBSD anymore. Stay tuned!
Reggle 
Hi.
After one month reading and looking info in internet(calomel.org,kernel-panic.it,openbsd faq,…) and many tries with different configutations,I decided to ask some help.
The issue is simple, I can’t match the outgoing traffic to carp ip address. When I go to some “show myip” web, it always appears the pysical one. Never the carp one.
As my ISP provider gives us 4 ips, I use two (one for each nic of the firewalls connected to internet) for physical interfaces and the other two for the carp interfaces.
Internet
|
+———————–/ \——————–+
| carp13(em0) = 81.92.37.13 |
| carp12(em0) = 81.92.37.12 |
| |
81.92.37.10 bge0 bge0 81.92.37.11
| |
+——-+ 10.0.0.2 +——-+
| fw0 |- re0 —- CARP Pfsync — re0 -| fw1 |
+——-+ 10.0.0.1 +——-+
| |
192.168.28.1 bge1 bge1 192.169.28.2
| |
| carp28(em1) = 192.168.28.11 |
+———————-\ /———————+
|
Internal LAN
(192.168.28/24)
Config files(from one of the two firewalls):
hostname.bge0
inet 81.92.37.10 255.255.255.248 NONE
hostname.bge1
inet 192.168.28.1 255.255.255.0 NONE
hostname.carp12
inet 81.92.37.12 255.255.255.248 81.92.37.15 balancing ip carpnodes 120:0,121:100 pass PaSSWord12
hostname.carp13
inet 81.92.37.12 255.255.255.248 81.92.37.15 balancing ip carpnodes 130:100,131:0 pass PaSSWord13
hostname.carp28
inet 192.168.28.11 255.255.255.0 192.168.28.255 balancing ip carpnodes 28:0,29:100 pass PaSSWord28
hostname.re0
inet 10.0.0.1 255.255.255.0 NONE
hostname.pfsync0
up syncdev re0
sysctl.conf
net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4 packets
net.inet.icmp.rediraccept=1 # 1=Accept ICMP redirects
net.inet.carp.preempt=1 # 1=Enable carp(4) preemption
net.inet.carp.log=3 # log level of carp(4) info, default 2
net.inet.carp.allow=1
net.inet.carp.arpbalance=0
pf.conf
priv_nets= “{127.0.0.0/8, 172.16.0.0/12, 10.0.0.0/8}”
set block-policy drop
set skip on lo
#Nat outgoing connections
match out on $ext_IF inet from !$ext_IF to any nat-to $ext_IF
#This does not work: match out on $ext_IF inet from !$ext_IF to any nat-to carp12
#Filter rules
block log all
block in quick from urpf-failed #spoofed address protection
#packet normaliztion
match in all scrub (no-df)
pass quick log on re0 inet proto pfsync keep state (no-sync)
pass in quick log on $ext_IF proto carp from carp12 to 224.0.0.18 keep state
pass in quick log on $ext_IF proto carp from carp13 to 224.0.0.18 keep state
pass in quick log on $int_IF proto carp from carp28 to 224.0.0.18 keep state
pass on { $int_IF , $ext_IF } proto carp keep state (no-sync)
block in quick on $ext_IF from $priv_nets to any
block out quick on $ext_IF from any to $priv_nets
block quick inet6
I’m not an expert, but I know this: CARP, as any other redundancy protocol, only serves as a virtual gateway for receiving traffic, not sending. So return traffic and traffic from the outside will be sent from the physical interface. That’s the way routing works, and is the same for CARP, VRRP, HRSP and GLBP. There is no way to change that behaviour as far as I know.
Normally, gateway redundancy is only used inside towards multiple ISP-uplinks in case of a larger network. Using gateway redundancy on outside interfaces is not commonly done, though combined with DNS records pointing to the virtual IP may provide some redundancy to a web server, if set up right.
Hi Reggle,
Your statement is correct but there is a solution…
As far as PF is concerned, network traffic comes from the physical interface, not the CARP virtual interface (i.e., carp0). So, write your rule sets accordingly. Don’t forget that an interface name in a PF rule can be either the name of a physical interface or an address associated with that interface.
For example, this rule could be correct:
pass in on bge0 inet proto tcp from any to carp0 port 22
I am afraid Reggle is half correct here, and you are half right too.
i’ve been figting like a mad man to create a nat-to rule to sent the traffic via my carp1 interface(external).
from any directly connected network (i have carp0 and carp2) the behavior is correct and the packet exit and come back via carp1 and goes back to the source.
when the packets have a source other than a directly connected network (the route of said is learned via a routing protocol), it happily throws my packet via the physical interface, wether I use nat-to carp1 nat-to $external_ip… the packet comes back on carp1 but never goes back to carp2 (state was on em1 not carp1)
so close yet so far…
forget my last comment that I wrote in a half insane state of mind at 3:00am
Reggle is correct saying the traffic is sent via the physical interface (even if said interface have no ip).
RACK is also correct saying that you can nat-to the carpX ip, Openbsd will keep the state on the physical interface, carp will receive the packet and the physical interface will allow it.
(10.10.10.253 = carp1 ip)
(192.168.14.2 = client from remote network present in openbsd routing table)
pfctl -vvv -s states | grep -A 2 192.168.14.2
em1 tcp 10.10.10.253:64184 (192.168.14.2:54295) -> 74.125.226.152:80 FIN_WAIT_2:FIN_WAIT_2
[1851812922 + 42624] wscale 7 [2750321222 + 30464] wscale 7
age 00:00:21, expires in 00:01:10, 6:4 pkts, 427:790 bytes, rule 8
So the solution for Paquitiu would be to have the proper nat-to carpX in his ruleset.
a few exaples here (notice the rule is pass out on physical interface and nat-to, carp0:network can be replaced by ip/netmask and carp1 can be replaced by the ip of carp1)
#nat inside network to em1 ip
pass out on em1 inet from carp0:network to any nat-to carp1