Efficient connection logging on a switch.

I know, it’s been quiet on this blog for the past months. But here we are again, starting off with a simple post. Maybe not much real world practical use, but fun to know.

Dealing with ACLs requires more protocol knowledge compared to dealing with a stateful firewall. A stateful firewall takes care of return traffic for you, and often even has some higher layer functionality so it can automatically allow the incoming port of a FTP connection, for example.

ACLs don’t do this. They’re static and don’t care for return traffic. On a switch in particular, they’re done on the ASIC, in hardware. This means there is no logging possible. On the plus side, the filtering doesn’t consume CPU. Many engineers assume stateful firewalls are superior to ACLs, and while this is certainly true concerning scalability and manageability, it’s not 100% true for security. ACLs don’t get fooled by some attacks: they’re not dynamic like the stateful filtering principle, so they will also not work for attacks that try to use the stateful functionality of a firewall against it. Attacks involving many packets attempting to use op CPU also don’t work. True, these attacks are not common, but it’s still a forgotten advantage.

Concerning logging, an ACL does have the option, even on a switch. By adding the ‘log’ parameter to the end of a line you can count the hits on the ACL. However, all this does on the ASIC is punt the packet towards the CPU, who then processes the packet in software and increases a counter and logs a syslog message. If you do this for a lot of packets, or all of them, you’re essentially using the CPU for switching. This defeats the purpose of the ASIC. Most Cisco switch don’t have the CPU for that, limiting throughput and causing high CPU, latency, jitter, even packetloss.

But there’s a more efficient way to do this, for TCP at least: only log the connection initiations.

permit tcp any any established
permit tcp any any eq 80
permit tcp any any eq 443
permit tcp any any eq 22 log
permit udp any any
deny ip any any

In the above ACL the first line will allow all TCP traffic for which there already is a connection established. Just this rule doesn’t allow any traffic: you still need to be able to initiate traffic too. The two rules below it allow HTTP and HTTPS. Finally, the fourth rule allows SSH but also logs it. When a user creates an SSH session through the interface, the first SYN packet will match the log rule and the connection will be logged. All packets of the same flow after this will match the first rule and be switched in hardware. Minimal CPU strain, but connections are logged. Apart from the first packet the flow is forwarded in hardware.

This way of ACL building also has advantages on CPU-based platforms like (low-end) routers: most packets will hit the first line of the ACL and only a few packets will require multiple ACL entries to be checked by the CPU, decreasing general load.

Using ACLs to mimic a firewall.

A stateful firewall should be considered a mandatory part of any network design. End user systems and the internet simply cannot be trusted, and even on servers there’s always the unexpected open port possibility. Going completely Cisco here for a moment, it’s an ASA, or a Zone-Based Firewall (ZBFW) configuration on an edge router.

But, usually due to budget constraints, it’s not always feasible to go and put firewalls everywhere in the network. While the WAN or internet edge should really have it, internally it can be less of a need. This is usually where access control lists (ACLs) come into play.

The example LAN above is completely internal. The switches are layer 2 access switches, the router depicted in the middle is a layer 3 switch acting as a default gateway for the VLANs.

Now let’s assume there’s a web server on 10.3.32.2 and a Voice server on 10.3.32.3. You want the following rules applied:

• From Clients access to the web server by http.
• From the IP Phones access to the Voice server using SIP protocol.
• Allow IP Phones to be pinged from the Voice server.
• Deny everything else.

On an ASA or any other stateful firewall, this would be a rule specified on VLAN 10, a rule for Voice on VLAN 20 and a rule for ICMP echo on VLAN 800. The stateful part would take care of the rest and automatically allow return traffic for existing connections. For TCP, it does so by checking the three-way handshake (SYN, SYN/ACK, ACK) and checking the connection breakdown (FIN and RST flags). For UDP, most stateful firewalls use a pseudo-stateful behaviour: the first UDP packet in a permitted rule will be set in the state table, with a time-out. Return traffic is accepted as long as the time-out isn’t reached. For each outgoing UDP packet part of the same stream, the timer is reset to zero. This is usually sufficient for connectionless communication.

ACLs however simply perform filtering and do not keep track of sessions. To do a ‘deny everything else’, both incoming and outgoing connections will have to be filtered properly with an ACL. A start would be this:

ip access-list extended VLAN10-IN
permit ip any host 10.3.32.2
ip access-list extended VLAN20-IN
permit ip any host 10.3.32.3
ip access-list extended VLAN30-IN
permit ip host 10.3.32.2 10.0.10.0 0.0.0.255
permit ip host 10.3.32.3 10.0.20.0 0.0.0.255

But the above access-list is very general and leaves a lot of security issues. To make it mimic a stateful firewall, add more details:

ip access-list extended VLAN10-IN
permit tcp 10.0.10.0 0.0.0.255 host 10.3.32.2 eq 80
ip access-list extended VLAN20-IN
permit udp 10.0.20.0 0.0.0.255 host 10.3.32.3 eq 5060
permit icmp 10.0.20.0 0.0.0.255 host 10.3.32.3 echo-reply
ip access-list extended VLAN30-IN
permit tcp host 10.3.32.2 eq 80 10.0.10.0 0.0.0.255 established
permit udp host 10.3.32.3 eq 5060 10.0.20.0 0.0.0.255
permit icmp host 10.3.32.3 10.0.20.0 0.0.0.255 echo

A breakdown of the above added parameters:

• Specifying a subnet with wildcard rather than just ‘any’, even if there’s just one subnet behind that interface, prevents IP spoofing. If just the return traffic is filtered on the subnet, this still allows for initial SYN packets, so SYN flooding from a spoofed IP is possible when ‘any’ is used.
• Always specify a destination port to prevent scanning of the servers and accidental (or malicious) connection to other services running on that server. Even if the server runs a software firewall: it shares the operating system with the services, who may open ports on this firewall.
• ICMP: specific definition of echo and echo-reply makes sure the server can ping the IP Phones, but not the other way around.
• In the return traffic, setting the source port to the service makes sure the server does not initiate connections, or in the case of UDP, starts a flow on a non-standard port. Since it’s not possible to use timers in the ACL based on outgoing connections, this way you can still do some filtering.
• For TCP, the ‘established’ keyword only allows packets that have the ACK bit set. The initial SYN used for the connection buildup is not allowed – Effectively preventing the server from initiating a connection himself. Instead, he has to listen for connections from the clients, where the ACK bit isn’t checked.

While this does not create a state table on the switch or router, it really narrows down the attack surface. Connections cannot be initiated where you don’t want it, packets must go to and originate from ports you decide. The biggest security problem is that UDP packets or TCP packets with ACK already set can be used to flood the links, but even then only from certain ports. And, compared to a stateful firewall, this configuration is more complex, because with each rule change, you have to completely understand the flow, and also adapt the ACL for the return traffic.

IPv6 RA Guard feature.

As promised an IPv6 post! The Cisco IPv6 workshop was a real eye-opener for me, with plenty of configuration examples, best-practices and highlighting important security issues. Something that had been lingering on my mind for some time was the fact that, in my IPv6 experiments so far, any device could send out a Router Advertisement (RA) if it wanted to, and it would automatically become the gateway for the subnet. No questions asked. Since I’m running my IPv6 router and tunnel on a virtual machine using 50 MB RAM, I became increasingly worried about the simplicity of a possible man-in-the-middle attack. Just set up a VM in a campus LAN, make it send out RAs, and capture all traffic. IPv6 is even preferred above IPv4 when a modern OS has the choice. That’s a big security hole which is hard to detect.

Turns out, when I asked my question, that Cisco had an answer ready for me: RA Guard, which works similar to DHCP Snooping for IPv4. RA Guard filters out any ICMPv6 type 134 messages (RA) coming from any port that is marked as untrusted. For the moment it seems to be available on Catalyst 6500 Series only, since software release 12.2(33)SXI4. A full configuration guide for various IPv6 security measures can be found on the Cisco website.

I was told that support for other platforms would eventually be possible. For now, a workaround exists using IPv6 access lists, but this was not recommended I was told, as this would be done in software, not in hardware. Nevertheless, since the issue exists in a present day network already, I tried configuring it.

Here’s the configuration, which uses a Port ACL:

WS-C3560-8PC#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
WS-C3560-8PC(config)#ipv6 access-list RA-GUARD
WS-C3560-8PC(config-ipv6-acl)#sequence 3 deny icmp any any router-advertisement
WS-C3560-8PC(config-ipv6-acl)#sequence 6 permit ipv6 any any
WS-C3560-8PC(config-ipv6-acl)#exit
WS-C3560-8PC(config)#interface FastEthernet0/5
WS-C3560-8PC(config-if)#ipv6 traffic-filter RA-GUARD in
WS-C3560-8PC#show ipv6 access-list
IPv6 access list RA-GUARD
deny icmp any any  router-advertisement sequence 3
permit ipv6 any any (24 matches) sequence 6

Quite simple and it turns out it works. FastEthernet0/5 had an IPv6 router behind it, sending RAs every 60 seconds. Strange fact: all IPv6 traffic was matched against the permit ACL as expected, but the RAs weren’t matched against the deny statement, nor the permit statement. No logging to be found, though the switch did drop them. My computers did not detect any IPv6 router. Disabling the traffic-filter and waiting a minute made the clients configure an automatic IPv6 address.

Unfortunately, I couldn’t do any throughput tests, so I can’t tell what the impact on CPU is, but if rogue RAs are really an issue in the network, it might be worth the increased CPU.

IPv6 firewalling on Vyatta.

A while ago, I described how to set up an IPv6 Tunnel, so you could experiment with IPv6 at home. But while I’ve set it up myself a dozen times by now, I rarely had the tunnel active for a long time. Reason: I’m not too happy about a connection without any sort of firewall applied. So in this post, I’m going to describe how to set up a basic IPv6 firewall on an interface in Vyatta. The Cisco-based solution might follow later. To be honest, I’m not even going to try the Windows Server based solution.

Vyatta allows for three directions to apply the rules: in, out and local. Local means traffic destined for the device itself. This is important: I do not have to define any entries in my rules concerning local packets, like routing protocols, LLDP and IPv6 neighbour discovery and router solicitation.

Also, I cannot just block all ICMPv6 packets. IPv6 routers cannot perform fragmentation, unlike IPv4. This means fragmentation has to be done by end nodes. An ICMPv6 type 2 datagram, “Packet too big”, has to be allowed through, otherwise connections will fail if a too big MTU value is used (because the end nodes never find out why it doesn’t work). ICMPv6 type 3 code 1, “Time exceeded” packets tell when fragments are lost during transit.  Finally, types 133-136 are for router solicitation and neighbour discovery, but are handled as local on the Vyatta. If you’re using other platforms, be aware that these types need to be allowed too.

Now for the real ‘firewalling’ part. I’m going to allow echo replies in, so I can ping outside devices. I will not be allowing echo requests in. Yes, that’s making me feel more secure, although it doesn’t have any added value anymore. I’m also letting IPSec through because I’m doing VPN experiments later on. Apart from that, UDP will be allowed, and TCP packets with the ACK bit set. This is similar to the keyword ‘established’ on a Cisco ACL, and will only allow return packets of active sessions. This is not the same as a stateful firewall, but it provides a basic defense against connection attempts. Here is the code:

reggle@vyatta# set firewall ipv6-name WANFW
[edit]
reggle@vyatta# set firewall ipv6-name WANFW description “Firewall to block incoming connections from IPv6 Tunnel”
[edit]
reggle@vyatta# set firewall ipv6-name WANFW default-action drop
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 5 protocol icmpv6
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 5 icmpv6 type packet-too-big
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 5 action accept
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 5 description “Must be allowed or MTU discovery will break”
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 10 protocol icmpv6
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 10 icmpv6 type pong
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 10 action accept
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 10 description “Allow ping replies”
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 15 protocol icmpv6
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 15 icmpv6 type time-exceeded
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 15 action accept
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 15 description “May cause fragmentation issues otherwise”
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 20 ipsec match-ipsec
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 20 action accept
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 20 description “Allow incoming IPSec connections”
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 30 protocol tcp
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 30 tcp flags ACK
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 30 action accept
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 30 description “Allow established TCP connections”
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 35 protocol udp
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 35 action accept
[edit]
reggle@vyatta# set firewall ipv6-name WANFW rule 35 description “Allow stateless UDP”
[edit]

That’s it. After that, do a ‘set interfaces tunnel tun0 firewall in ipv6-name WANFW’, followed by ‘commit’, and it should work. You can test this by trying to ping an IPv6 host (like ipv6.google.com), which should work (echo reply allowed), while a ping to your computer should fail (echo request not allowed). The latter you can test on this website. Don’t forget to save the configuration if it works!

Hardening switch security.

Another day, another thing to do in my network. I’m currently taking the first steps of increasing my network’s security and preparing for the deployment of a permanent VPN server.

First the VPN server: I’ve already done tests, but I ran into common subnet problems. The current subnet I use is 192.168.0.0/24, and from several other (friend’s) places, connecting works but routing fails, since their local network also uses 192.168.0.0/24 very often. So I’ve migrated my subnet to something less common, e.g. 192.168.168.0/24 to reduce complexity. More on VPN in blog posts to come, when I properly figure out the details.

Second issue: I got tired of typing all those IP addresses over and over everytime I tested or changed something. It’s possible to set up an internal DNS to take care of that, but that’s still on the ‘to do’ list, so for now, I’m using the hosts file. For those who don’t know what the hosts file is: it’s a file in Windows that contains DNS records, in the “C:/Windows/system32/drivers/etc/” directory. It usually contains no records because everything is done by DNS server, but you can add your own records to it. In this case, for example, I added my IP Phone: ‘192.168.168.161 phone’. Now if I type ‘ping phone’ in the command line, it will ping my IP Phone (which of course has a static IP). Might be useful if you have some devices around the house with a static IP but no DNS server. Naturally, it will only work from that computer.

And last: hardening my switch security. I originally accessed my switch remotely by Telnet on port 23. This is about the most insecure thing to do. Time to implement SSH (RFC 4251 for those interested) and access-lists (ACL). First ACL: I want to be able to access the switch, but only from certain locations: the inside network and my workplace. At my office I’m connected behind a NAT device, but this device’s external IP is static, so that makes it simple:

Switch(config)#ip access-list standard remoteACL
Switch(config-std-acl)#10 permit host <ip office>
Switch(config-std-acl)#20 permit 192.168.168.0 0.0.0.255
Switch(config-std-acl)#30 deny any log
Switch(config-std-acl)#exit

The first line is from my work, the second for when I’m at home, and the third line is just to log any unauthorized attempts. The third is optional because of the implicit deny at the end. Next, I configure usernames to log in, and activate aaa services needed for usernames and SSH.

Switch(config)#aaa new-model
Switch(config)#username <user> privilege 15 secret <password>
Switch(config)#aaa authentication login default local
Switch(config)#aaa authentication enable default none

Now, when logging in remotely, I will be asked a username and password. The privilege level is set to 15 because this is the highest level (all rights), and the last two commands are to make sure the device will look at the local user database when a login attempt is made. You can also use an external database using TACACS+ or RADIUS, pointing to a database server, e.g. Windows Active Directory. Not needed for the lab, but again, recommended for use in a company.

Next, the actual SSH configuration. This requires an IOS that supports cryptographic services for this (look for ‘k9’ in the name). You will need to generate a RSA key, but for that key, you will need a domain name, and the device must have a non-default hostname.

Switch(config)#hostname switch01
switch01(config)#ip domain-name local.lan
switch01(config)#crypto key generate rsa

Now you will be asked a key length. Standard is 512 bits, but it’s best to make is more secure using 1024 bits. Many sources, including Cisco, recommend this as a minimum length. The longer the key length the more time it takes to generate. Once the key is generated, your device has a valid certificate. If you change the hostname or domain name afterwards, you can still connect, but the key will be marked as untrusted by any properly working client program. Also, I’m using local certificates, but when managing many devices in the same company, it’s best to set up a central certificate. Yes, that’s on my ‘to do’ list too. Once the key is generated, we apply the configuration. We make sure only the more recent SSHv2 is used, not SSHv1 or Telnet. Also, you may have noticed I didn’t apply the ACL above yet, so I’m doing that too now.

switch01(config)#ip ssh version 2
switch01(config)#line vty 0 15
switch01(config-line)#access-class remoteACL in
switch01(config-line)#transport input ssh

Now we can connect using SSH on port 22. The only thing I would still want to change is the port number. Reason for this is that most automated scanning tools and most attempted unauthorized logins to systems are done on standard ports. Cisco published a very interesting white paper about this, really worth the read. Problem is that my switch does not seem to support these commands, despite having a crypto-IOS. It seems this is only for routers. I’m listing the commands anyway:

Router(config)#ip ssh port 2000 rotary 1
Router(config)#line vty 0 4
Router(config-line)#rotary 1

SSH will now listen on port 2000 as well. Mind the ‘as well’: it still listens on port 22. You’ll need to define an extended ACL with port numbers to deny connections on port 22. Alternatively, you can set op a static port translation in your home router, from port 2000 to port 22. This would be much easier, but my ISP’s router does not permit it, so I’m stuck.

Either way, my security has increased now. Before, anybody could log in on my switch using a password they may have captured earlier when I connected using Telnet, which sends everything in clear text. Now, everything is encrypted, and the ACL restricts connections, so it only works from my workplace.