The new 3850 switch, UPOE and compact switches: interesting design options.

Initially, I told myself I wasn’t going to spend an entire blog post to the new 3850 switch. Presented on Cisco Live London, it was just a new series of Catalyst switches with all existing functionality plus some new things like the integrated WLC. Not really interested in anything else, I got bored and Googled around to see if my WS-C3560-8PC fanless desktop switch had a gigabit equivalent. Turns out it does, with quite a few models. All fanless, PoE capabilities and some layer 3 models. But then I noticed something: some of these models had both PoE powering and PoE powered ports. So how does that work? Well, these switches have two PoE powered ports. Some power is used for the switch itself and the remainder is used in the PoE budget towards the switches. The switches can give standard 802.3af PoE at 15.4W per port, but can receive 802.3af (15.4W), 802.3at PoE+ (25,5W) or UPoE. UPoE isn’t an official standard yet, but Cisco is pushing for it and their new 3850 series supports it. It allows up to 60 Watts per port.

And then it hit me. 3850’s, compact switches, and other PoE devices like Access Points and IP Phones allow for a lot of interesting designs.

For example, using the 3850 in a stack as a core switch in an office, it’s possible to give power to fanless (silent) compact switches near the desks. The C2960CPD-8PT (what a name!) has two powered PoE uplinks so a 2 Gbps port-channel towards the 3850 stack is possible. That gives redundancy when spread over two switches, and the stack can also use StackPower. It’s max power budget is 30W, which is enough for two PoE devices, or more when using 802.3af classes or CDP PoE negotiation, e.g. many IP Phones only use 7 Watt, allowing for four on one switch. Result: it’s possible to power an entire office network directly from the core: less wall sockets needed, less cable runs. And finally, the integrated WLC on the 3850 allows all AP tunnels to terminate in the office itself, instead of going back towards a remote controller in a HQ/Data Center. This allows faster and easier access to local resources (printers, on-site servers).

The combination 3850/Compact switches allows for long cable runs without any need for a power supply anywhere in between. In the most extreme case you can bridge 500 meters with it: 3850, 100m copper  UPoE, 2960CPD, 100m PoE, 2960C, and a 100m cable run towards a mirrored setup.

So yes, when thinking about it, these new switches with new technology do allow more flexibility when deploying a network!

Advertisement

Cisco Live 2013: a great experience.

Last week a colleague and I went to Cisco Live London 2013! Overall experience: great. I gained an impressive amount of knowledge in a short time, and writing all that down would generate enough blog posts for a year. We went to many different seminars that really went into the details. Too much to name, but a quick review of the impressive stuff I saw:

• The launch of the Catalyst 3850 switch. I’m not going to read the data sheet, but quick overview: all 3750X functionality (StackWise, StackPower, hardware IPv6, QoS, redundant power and fans), 802.3at PoE+, 32k CAM table possible, integrated Wireless LAN Controller, up to 40 Gbps WLC throughput, and besides the 24 or 48 Gigabit ports up to 4x 10GE is possible in a SFP+ module. Very promising, although most functionality is expected for a next-generation switch, and I have yet to test it all in a production environment.
• A seminar about the Nexus 6000 switch. Designed towards ultra low-latency implementations. While I did think initially that it would be of little use cost/benefits-wise outside of the trading and high computing markets, an article by Greg Ferro made me think about this again. Apparently he was there too in London. Worth the read.
• The Nexus 7000 architecture seminar. This was a bit of a disappointment. Cisco’s flagship data center switch sounds like an elephant: huge, strong and loudly announcing it’s there, but too many features that aren’t ready yet, so watch out what you want to do with it. I noticed plenty of limitations on the current generation that will probably go unnoticed if you’re not using the Nexus to its full potential, but no one buys a Nexus 7000 just for basic switching.
• Energywise fundamentals and deployment by John Parello. An unexpectedly useful technology, that I have working now (more about that in a later blog post). I find that this requires more attention and research.
• FCoE practical lab: a complete overview of the technology, much more in-dept than my own brief review. Again an interesting technology that will prove useful when mastered. For practical implementation it requires a lot of planning and design in advance though.
• Ultra Low Latency Data Center design: I already mentioned some details on latency in my article about fibers. Though not that important for me in practice, I learned some key points about latency.
• Multicast troubleshooting: Luc De Ghein did a great explanation there, giving me insight into multicast inner workings.
• Meet The Engineer meeting with Lars, an engineer sharing the same native language as I did who is a Catalyst 6500 switch expert. He provided me good insights with hardware and software ACLs and Control Plane Policing.
• CCIE Troubleshooting lab: me and my colleague actually managed to solve part of it… But the 45 minutes time was just too short. A nice taste of things that may come for me one day.

For a network engineer this was a really great event, much to see, to learn, to experience. Probably a once-in-a-lifetime chance for me, and I don’t regret it.

Loop prevention and limitation.

When dealing with large layer 2 broadcast domains, loop prevention becomes one of your main concerns. Loops cause broadcast storms, and broadcast storms bring down networks.

But loop prevention in a data center is not simple. For one, you’re not in control of all devices capable of forming a loop. You’re not even in control of every device capable of generating spanning tree BPDU’s. And then there are vSwitches, which are switches but don’t use spanning-tree (an in-dept discussion of this can be found on Ivan Pepelnjak’s blog, here and here). To finish it off, Nexus 2000 Fabric EXtenders have BPDU Guard hardcoded, although I’ve been informed this might change.

Despite the above gaps in uniformity for spanning-tree, it’s still considered mandatory to run it on the network. Without it, things would be much worse and a proper spanning-tree design with an appropriate type (MST, RPVST) can prevent most issues. BPDU Guard towards (non-hypervisor) servers is a second rule of thumb.

But BPDU Guard only activates upon receiving a BPDU, and loops don’t always carry BPDU’s. This is where storm-control comes in the picture: it checks the number of broadcast, multicast and unknown unicast frames over a 1 second period. If it reaches a configured threshold, any following offending packets are dropped in that 1 second interval. Optionally, the port can be err-disabled, and snmp traps can be sent. Configuration is as following:

Switch(config-if)#storm-control broadcast level pps 100
Switch(config-if)#storm-control multicast level 2.00
Switch(config-if)#storm-control unicast level bps 1m
Switch(config-if)#storm-control action shutdown
Switch(config-if)#storm-control action trap

This is just a sample config: broadcast, multicast and unknown unicast can be configured separately. The threshold can be expressed in percentage of total bandwidth (2.00), packets per second (pps 100) and bits per second (bps 1m, or 1,000,000). There is no real guideline for good values, because it depends on your network. The more you know which traffic flows through it, the better you can set a good value. It’s not a perfect solution, but it’s a measure of last resort for when a loop does occur. The ‘shutdown’ action should only be configured on the same links where you would want BPDU Guard.

So are inter-switch links safe then? Unfortunately not, due to the architecture of switches that forward in hardware.

The hardware forwarding uses ASICs that only know how to forward. This is referred to as the data plane. The CPU does not do any forwarding (well, it can, but it shouldn’t), and is referred to as the control plane. It monitors the switch health, processes BPDUs, CDP frames, and optionally for a layer 3 switch, the routing protocols and the routing table (but again, not the actual layer 3 forwarding). If the control plane takes a hit and crashes (bug, exploit, sometimes hardware), the control plane will crash. But chances are the data plane will not. And without control plane to limit it, the data plane will happily start forwarding any frame entering the switch out of every other port, regardless of VLAN, CAM table or layer 3 headers. Even ports configured as layer 3 will start flooding.

You might notice ICMP replies mentioned in the picture. This can be an attack vector on a layer 3 switch: generating many frames with low TTL will cause it to generate ICMP TTL Expired messages. Combined with ping replies and incoming packets for non-existing hosts in connected subnets, which will generate ARP requests, this can burden a switch CPU beyond normal expectations.

When this happens, storm control with SNMP traps is one of your best options to quickly finding and limiting the problem on a large layer 2 domain. Another option in the initial design is using actual routers for layer 3 boundaries if possible: the interfaces are hardcoded layer 3 and will not start flooding when the control plane crashes.

Private VLANs over multiple devices.

I’ve discussed Private VLANs before and I still consider them a useful technology. However, most mentions and implementations of PVLANs only use one switch, and it’s not clear what the interactions of PVLANs with other devices are.

The above setup features three switches and a firewall, all connected through trunk links. SW1 and SW3 can use PVLANs, and SW2 can use VLANs but has no awareness of Private VLANs, as it’s a transit switch. The firewall acts as default gateway for all the VLANs, including the PVLANs (a real-world example would be an ASA 5510 here). The requirements of the setup are that an isolated port on one switch can’t talk with an isolated port on another switch, an isolated port on one switch can talk with a promiscuous port on another switch, it works over transit switches, and the trunk link can be used as default gateway. Will this scenario work? Let’s examine.

Cisco recommends to define both the isolated and promiscuous VLANs on all trunk links and transit switches. To remain consistent between switches, frames are always sent over trunk links tagged with the originating VLAN. A frame coming from an isolated VLAN will go over a trunk link tagged as the isolated VLAN. This way, the next switch knows it came from an isolated port and will only allow communication with either a promiscuous port or another trunk link that allows the isolated VLAN.

This logic solves the first requirements: the information of the origin port is carried over to the next switch, and PVLANs can work perfectly over multiple switches. Does this also include the unaware transit switch? Yes it does: it treats the PVLANs as just a VLAN and allows traffic over the trunk links. Of course, if the switch is not aware of Private VLANs, it’s not possible to configure access ports in those VLANs, isolated or promiscuous.

But how does return traffic behave? If a frame is sent from an isolated port to a promiscuous one (a default gateway), how does it return? It returns tagged as the promiscuous VLAN over the trunk links and can communicate to all ports: trunk , isolated and promiscuous ports. And this is where the flaw in the PVLAN design is: over trunk links, the switch expects return traffic to arrive over another VLAN. A ping from a device towards the default gateway goes over the trunks in the isolated VLAN and returns in the promiscuous VLAN. And a firewall with trunk links (e.g. an ASA) can’t handle this: it has no awareness of Private VLANs. It will send return traffic down the same VLAN, and once this arrives on the switch, the switch will think it’s a frame from an isolated VLAN access port and deny communication with all other isolated ports. Return traffic cannot reach the destination anymore.

In defence of  the ASA, not a single vendor has PVLAN awareness in firewalls or routers. And this cripples the technology in larger environments, as you’re constrained to use one access port per VLAN in a Private VLAN environment, and this doesn’t scale when dealing with many VLANs.

Update 13/03/2014: the test setup for this blog post did not cover MAC address learning so I missed a very important fact: non-PVLAN aware switches will flood traffic in the Private VLANs, as they will never learn the destination MAC addresses of traffic in the same VLAN. This will not scale. Thanks to the networking-forum.net members again for the nudge in the right direction!

Dynamic Trunk Protocol: Wireshark research.

Time to review something that has been very unclear to me from the beginning: Dynamic Trunking Protocol. First off, it’s a Cisco-only protocol, and let’s be honest, it has no real world uses except acting as a security risk. Second: most literature about it is in fact quite unclear, except the CCIE OCG so far, but even then I’m left with question marks.

So let’s go through the theory first, as this is usually the best way to start:

• DTP sends special frames out of every switchport by default, which can be used to negotiate a trunk link, and negotiate the encapsulation (802.1q or Cisco-proprietary ISL).
• The frames can either be ‘auto’, ‘desirable’, or ‘on’. While ‘desirable’ and ‘on’ actively try to negotiate a trunk, ‘auto’ only negotiates one when the other end is set to trunking.
• The ‘switchport nonegotiate’ command prevents DTP frames from being sent.

Now, let’s see how it goes in practice.

Commands without effect
It’s never mentioned explicitly in any literature I’ve encountered so far, so I tested it just to be sure. The commands ‘switchport access vlan number‘ and ‘switchport trunk allowed vlan numbers‘ do not have any effect on DTP at all. Should the port become an access port, then the ‘switchport access’ command is used to define the VLAN, and the trunk allowed list is ignored. Should the port become a trunk, the trunk allowed list is used and the access VLAN is ignored. Same for ‘switchport trunk native vlan number‘, which is only used should the port become a trunk.

Default configuration
The default configuration of a Cisco switch is to send out DTP auto frames. This means no trunk link will be formed unless configured to do so. And here’s the security risk: if you leave a port in it’s default configuration, someone sending a DTP desirable frame can form a trunk and gain access to all VLANs. In case you’re thinking ‘Oh, this is highly theoretical’: it took me less than 30 minutes to find, download and install Yersinia on a Linux VM and send DTP frames to my home lab switch.

Commands with effect, and their changes
So what actually influences DTP? Two commands, it seems: ‘switchport mode’, and ‘switchport nonegotiate’. I’m listing my findings below, after trying all possible combinations on a switch:

• Default configuration: DTP sends two frames every 30 seconds (a timer that can’t be changed): one untagged frame, one ISL encapsulated frame. Both are DTP ‘auto’ mode (status code 0x03). Since ISL has no concept of native VLAN and some (older) switches only support ISL, sending the second ISL frame makes sense for compatibility reasons. DTP also sends a DTP Type, which is a HEX code listing the supported encapsulation methods.
• ‘switchport mode access’: no DTP frames are sent. At all. The ‘switchport nonegotiate’ does not make any difference.
• ‘switchport mode trunk’: DTP frames are sent in mode ‘on’, with a status code of 0x81. Since you have to choose either ISL or 802.1q as an encapsulation method before you can make the port a trunk, DTP sends this parameter too: 0xa5 for 802.1q and 0x42 for ISL. Also, independent of the encapsulation chosen, DTP will again send two frames: one untagged, one ISL encapsulated.
  Note that this port will operate in trunk mode regardless of what the other end decides. The frames are sent to help the other end in negotiation and choosing the right encapsulation. A port with default configuration will convert to a trunk port when receiving these DTP frames.
  Using ‘switchport nonegotiate’ here stops the sending of DTP frames. Should this port connect to a port with default configuration now, it will still be a trunk, but the other and will stay an access port.
• ‘switchport mode dynamic auto’: this is the default configuration and this line will not even show in the running config.
• ‘switchport mode dynamic desirable’: again like the default configuration, the only difference being status code 0x04, and a trunk will actively form when a port running DTP is connected (no matter the other end’s DTP mode).

Another unexpected command which has effect is the ‘vtp domain name‘: I’ve researched that before.

Practical implications
So what’s best in practice to counter any unexpected behavior, and security risks?
In case of an access port, ‘switchport mode access’ is effective enough as it shuts down DTP on a port. ‘switchport nonegotiate’ is a redundant command, so it doesn’t matter whether it’s applied or not. Just defining ‘switchport access vlan number‘ without the matching ‘switchport mode access’ is a security risk.
In case of a trunk port, I would still recommend disabling DTP, so a static configuration with ‘switchport nonegotiate’ becomes mandatory.
And what if DTP somehow is required to run, can the risk be minimized? (This is theory, if someone requires DTP, you… try to stay calm.) Yes: ‘switchport trunk allowed vlan numbers‘ can be configured to only allow access to the default access VLAN, or even set to ‘none’. This command doesn’t become effective until a trunk is negotiated.
Last, the Nexus series doesn’t run DTP at all, which I personally regard as a huge plus.

So that’s my in-dept research. Probably nothing I will really need in real life, but good to know for the security consequences, as well as some more knowledge for the next certification.

BPDU Guard and BPDU Filter: the details.

I’m noticing a shift in focus in my articles: from theory to a more practical approach. This one, hopefully, is a mix of both: BPDU Guard and BPDU Filter.

Let’s start with the theory first: for those who don’t know, BPDU Guard is a feature on Cisco switches which causes a switchport to shut down as soon as it receives a spanning-tree frame. BPDU Filter doesn’t shut down the port, but instead filters out the BPDU, as if it was never received on that port. Both features can be configured globally on the switch, or on a per-port basis. Now to the practical side of things:

BPDU Guard
A port with BPDU Guard will still send out BPDU frames itself. This comes in handy, as two access ports with PortFast and BPDU Guard connected to each other in a Campus LAN will detect the BPDUs and shut down the port. This is a good practice, as you don’t always control what end users connect to the network: their own switches, hubs, computers with network cards placed in bridging mode… All of these aren’t a real problem until somehow a second connection is made. Using the command ‘spanning-tree portfast bpduguard default’, each PortFast port will automatically have it configured. You can even configure ‘errdisable recovery cause bpduguard’ and the switch will recover automatically from these incidents, no need for manual intervention.
But… There are downsides. First of all, the port isn’t shut down until an actual BPDU frame is received: since spanning-tree does this every two seconds by default, a broadcast frame can still loop for two seconds before the port is disabled. That might not be impressive, but it’s sometimes enough to briefly send a spike through the network. Also, while this does stop most loops made through IP Phones, I’ve heard people say some IP Phones filter out BPDU frames. And while the errdisable autorecovery saves management overhead, the loop will briefly reappear as long as the problematic cable isn’t removed.

BPDU Filter
BPDU Filter on a port filters out any received BPDU frames, but at the same time, it stops sending BPDUs as well. It effectively stops spanning-tree from operating on that port, so this port will stay enabled no matter what happens to the spanning-tree topology. Very tricky, as it’s always possible someone misplaces a cable (and we’re all human, so that will happen eventually).
Also… While ‘spanning-tree portfast bpdufilter default’ seems to enable it globally on the switch, it works different: it doesn’t send out BPDU frames on PortFast ports, but once a BPDU frame is received, the port loses PortFast status and starts working like a normal switchport, sending out BPDUs again.
Personally, I strongly dislike experimenting with spanning-tree in such a way, but in a rare occasion, it might be the only option. That rare occasion usually involves a Nexus 2000 Fabric Extender: the access ports are hardcoded with BPDU Guard enabled, so the only way to connect a switch to it is if that switch does not send out any BPDUs. A risky setup.

Any comments? Feel free to share the bridging loop horror stories!

Layer 2 design and security: use which features where?

Not every layer 2 design is the same. There are a lot of features and techniques you can use in a layer 2 LAN, but it really depends on the purpose of the LAN what is going to be effective and what not. Sometimes, it’s better to leave some features out because of unexpected consequences otherwise.

So far, in my experience, I’ve encountered four distinct types of layer 2 networks in practice.

The typical Campus LAN or office network is a network where mostly end users connect. In it’s most simple form, it’s one VLAN where the computers connect. As it grows, it will usually have a second VLAN for IP Phones, and if even larger, separate VLANs for different kinds of users, a separate VLAN for in-office services (think printers, content displaying media, perhaps security camera’s), and in case of a full-scale wireless architecture, a separate VLAN for Lightweight Access Points (LAPs). Typically, DHCP is going to be used a lot here, and users expect a ‘fast user experience’, which usually translates to low-latency, low- to medium bandwidth usage. Only rarely end users require full gigabit connectivity towards the desktop (although they usually think they do).

The following are typical design characteristics of such a Campus LAN:

• The typical access ports with optionally an auxiliary VLAN for Voice. Static configuration, perhaps dynamic VLAN assignment through 802.1x or other means if you’re up to the task.
• Things like ‘switchport nonegotiate’ and ‘no cdp enable’ should be obvious on these access ports. If Cisco IP Phones are used, CDP may be of use though.
• Interesting security features: DHCP Snooping (switch uplinks trusted), activated on client VLANs, port-security, BPDU Guard. Keep in mind port-security will count for any MAC address on any VLAN, so the IP Phone counts as one. Even setting the limit to 5 MAC addresses is better than not setting it at all, as it will counter any MAC exhaust attack.
• If you’re worried about having to go and re-enable a switchport every time BPDU Guard or port-security kicks in, you can configure err-disable recovery. If you don’t think that will happen at all, you have too much confidence in mankind.
• IP Phones require PoE and most models are capped at 100 Mbps, making a gigabit switch redundant if you daisy-chain computers behind the IP Phones. Personally, I like 100 Mbps to desktops in most situations, as applications don’t require more and it’s an easy way to limit one user from pulling too much bandwidth without configuring QoS.
• ARP Inspection, while certainly a good feature, may rarely not work correctly I’ve noticed. Still, a Campus LAN is the most likely place you’ll see an ARP Spoofing attack.
• Think dual stack. I’m going to stress my IPv6 RA Guard post once more to counter any IPv6-related attacks on the subnet. Blocking IP protocol 41 (IPv6 over IP) out of the network will counter any automatic tunneling mechanisms client devices may have (and Windows 7 has one configured by default).
• Taking the above in account, Cisco 2960 and 2960S series are usually perfect for this environment, with 3560v2 and 3750X should layer 3 switches be required.

A Server LAN is a bunch of physical servers connected to switches. A smaller company’s own Server LAN is often just one VLAN for all the servers. If there are internet-faced servers, like web servers or a proxy, they should have a dedicated ‘DMZ‘ VLAN, as these servers are most prone to direct hack attempts. Unlike the Campus LAN, high traffic volumes may occur in this here.

• At least gigabit is needed for a decent server, as multiple users will connect to one server. 100 Mbps is not forbidden, some services barely use bandwidth.
• DHCP Snooping and ARP Inspection are quite useless here. Servers have static IPs, and getting ARP Inspection working in such an environment requires a lot of static entries, configuration overhead, and difficult troubleshooting.
• The above mentioned RA Guard for IPv6 does stay valid, because of the different approach of IPv6. Use with care when used in software though.
• Port-security works, and can map a MAC address to a port. Servers don’t usually move in a physical environment, but in a virtualized environment with vMotion and the like, it’s of not much use.
• Things like ‘switchport nonegotiate’ and ‘no cdp enable’ should be obvious again.
• BPDU Guard, even on trunk links to servers, is a good idea. Some might argue that it’s not good to have an important server disconnected from the network because it happens to send out a BPDU frame by mistake, but I personally don’t consider that a network-related problem.
• Private VLANs can seriously increase security if deployed properly. It’s usually sufficient if the servers can communicate with the gateway. Doesn’t work if the servers need to see each other (a cluster heartbeat for example), and in virtualized environments, as it doesn’t work with VLAN tagging.
• If the budget allows it and you require QoS and bigger buffers, a Cisco 4948 becomes an interesting option.

A Data Center LAN is like a Server LAN, but heavily consolidated. Virtualization places many servers on one physical uplink. While a large company’s data center will not have a large number of VLANs, a colocation data center can have hundreds of VLANs, and even reach the maximum of 4096 VLANs in extreme cases.

• I consider gigabit mandatory, and 10 Gbps is becoming the standard these days. After all, several virtual servers share the link, and FCoE further consumes bandwidth.
• The remaining configuration is like a Server LAN, but because of the shared environment and a lot of trunk links, Private VLANs are not an option. DTP and CDP disabled on the server links, and BPDU Guard are the only usable security features.
• Again, IPv6 RA Guard, although I would recommend either IPv6 stack disabled, or configured static.
• QoS features would be recommended.
• Spanning-tree mode here should be MST. RPVST+ will generate many BPDUs that have to be handled in software.
• Data Center LAN requires data center switches. At least Cisco 4948, but this environment is the home of chassis switches, Cisco 4500, 6500, and the Nexus family.

The last layer 2 network is a core network. It’s an environment that does not do any filtering or functionality other than forwarding as fast as possible, e.g. a large Campus LAN core, or a provider backbone where BGP transit traffic passes.

• This is 10 Gbps or faster.
• As little extra functionality besides forwarding as possible, and if present, done in hardware.
• Cisco’s 6500 chassis has 10 Gbps blades, and even a new 4-port 40 Gbps blade: WS-X6904-40G-2T. Extreme Networks seems to have a more extended portfolio here, with the BlackDiamond X chassis claiming up to 192 40GbE ports.

This is just my opinion on things, a first combination of theoretical knowledge and field experience. If you don’t agree, let me know in the comments – I’m hoping for a discussion on this one.

A little bit of everything.

Yes, a bit of everything, that’s what it has been lately. First, I’m upgrading my home lab switches with more recent IOS versions. The 3560 on my desk can now run EIGRP for IPv6. My 2970 gigabit switch will follow tomorrow, with a K9 IOS this time to make it accessible via SSH.

Second is that I’ve been fine tuning my knowledge of layer 2 security features, using my 3560 desk switch and a 3750 test switch as subjects. RA Guard works great, and so does DHCP Snooping. DHCP Snooping has revealed a third functionality to me, next to countering rogue DHCP servers and preventing DHCP flooding: it also detects when a MAC address that sends an INFORM cannot be present on that port according to the mac address-table. It will generate a ‘%DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL’ message and drop the frame. Seems to be a functionality related to ARP Inspection.
And ARP Inspection, on the other hand, requires some planning of your DHCP servers: if multiple are present and they all reply at the same time, the DHCP Snooping feature, on which ARP Inspection relies, sometimes picks the wrong packet to add to it’s binding table. The client device selects another packet of the ones it received to configure itself, and thus ARP Inspection thinks there’s spoofing going on. I’m still figuring out how to effectively counter that.

Third is that I’ve ordered the CCIE Routing and Switching Certification Guide, 4th Edition hardcover, so I have a lot of reading to be done soon. I have to admit that I don’t like to read ebooks on a big screen so far, and I’m reluctant to buy a reader.
Yesterday I also tried a MPLS lab for the first time, with BGP-MP in GNS3. It did take me several hours but I managed to get it running. Not bad for never having done anything MPLS related before. Still, it’s a huge topic and I’ll need to learn a lot more about that.

And last, I tested an Aruba Remote Access Point (RAP). I’ve already tested Instant Access Points. The RAP works different: once booted, it needs an internet connection. When connecting a computer (it has LAN interfaces, just like a consumer-grade router), it redirects to a setup page, where you have to enter the public IP address of a Wireless LAN Controller (WLC). It then tries to negotiate a tunnel through NAT-T over UDP port 4500 to that WLC. It works by encapsulating IPsec in a UDP header, bypassing any NAT devices that are incapable of keeping the NAT state of IPsec.
The RAP tries to authenticate itself at the WLC using his MAC address. After whitelisting it and configuring a wireless profile (which contains the list of SSIDs to send out), I had to reboot the RAP. I ended up rebooting it several times, thinking it didn’t work, but eventually it turned out my cable had broken due to all the times I plugged it in and out again. The RAP booted fine and started sending out the correct SSIDs. Initially, the wireless connection didn’t hand out an IP to me, but after five minutes, everything suddenly got an IP and started working as if there had never been a problem. Not sure why this happened, although I suspect my NAT router of dropping some of the UDP packets (which wouldn’t be the first time).

A little bit of everything indeed.

IPv6 RA Guard feature.

As promised an IPv6 post! The Cisco IPv6 workshop was a real eye-opener for me, with plenty of configuration examples, best-practices and highlighting important security issues. Something that had been lingering on my mind for some time was the fact that, in my IPv6 experiments so far, any device could send out a Router Advertisement (RA) if it wanted to, and it would automatically become the gateway for the subnet. No questions asked. Since I’m running my IPv6 router and tunnel on a virtual machine using 50 MB RAM, I became increasingly worried about the simplicity of a possible man-in-the-middle attack. Just set up a VM in a campus LAN, make it send out RAs, and capture all traffic. IPv6 is even preferred above IPv4 when a modern OS has the choice. That’s a big security hole which is hard to detect.

Turns out, when I asked my question, that Cisco had an answer ready for me: RA Guard, which works similar to DHCP Snooping for IPv4. RA Guard filters out any ICMPv6 type 134 messages (RA) coming from any port that is marked as untrusted. For the moment it seems to be available on Catalyst 6500 Series only, since software release 12.2(33)SXI4. A full configuration guide for various IPv6 security measures can be found on the Cisco website.

I was told that support for other platforms would eventually be possible. For now, a workaround exists using IPv6 access lists, but this was not recommended I was told, as this would be done in software, not in hardware. Nevertheless, since the issue exists in a present day network already, I tried configuring it.

Here’s the configuration, which uses a Port ACL:

WS-C3560-8PC#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
WS-C3560-8PC(config)#ipv6 access-list RA-GUARD
WS-C3560-8PC(config-ipv6-acl)#sequence 3 deny icmp any any router-advertisement
WS-C3560-8PC(config-ipv6-acl)#sequence 6 permit ipv6 any any
WS-C3560-8PC(config-ipv6-acl)#exit
WS-C3560-8PC(config)#interface FastEthernet0/5
WS-C3560-8PC(config-if)#ipv6 traffic-filter RA-GUARD in
WS-C3560-8PC#show ipv6 access-list
IPv6 access list RA-GUARD
deny icmp any any  router-advertisement sequence 3
permit ipv6 any any (24 matches) sequence 6

Quite simple and it turns out it works. FastEthernet0/5 had an IPv6 router behind it, sending RAs every 60 seconds. Strange fact: all IPv6 traffic was matched against the permit ACL as expected, but the RAs weren’t matched against the deny statement, nor the permit statement. No logging to be found, though the switch did drop them. My computers did not detect any IPv6 router. Disabling the traffic-filter and waiting a minute made the clients configure an automatic IPv6 address.

Unfortunately, I couldn’t do any throughput tests, so I can’t tell what the impact on CPU is, but if rogue RAs are really an issue in the network, it might be worth the increased CPU.

Cisco’s StackWise and StackPower: useful extras if you know them.

Time to discuss some technologies I had the chance to work with: StackWise and StackPower.

Stackwise in theory

First StackWise. It is supported on the 3750 series of switches: 3750v2, 3750E and 3750X. On the rear side of these models, there are two stack ports. By connecting several switches together using these stack ports, one big logical switch is created. It’s possible to connect up to nine switches, and for once, you can do a ‘loop connect’ with the stack cables.

In each direction, a stack cable gives 16 Gbps bandwidth, making for a total bandwidth of 32 Gbps. Since the stack creates one logical switch, the stack ‘ring’ does not participate in spanning-tree and no stack port is ever put in blocking state, and the stack is manageable through one single management IP. Yes, it shows a whopping 432 ports for a stack of nine 48-port switches. Another benefit is that port-channels can be terminated over multiple physical switches in the same stack (I’ve already written about that).

The switches choose a stack master among themselves. This one is the ‘master’ switch, in control of everything. Master switches are chosen on various factors, but in practice, switch priority (configurable) and the first to boot are usually key to the master election process.

The 3750E and 3750X series support StackWise Plus. This is an enchancement on StackWise with better switching decisions, making less intensive use of the stack cable bandwidth. Cisco claims up to 64 Gbps backplane switching with this, but I haven’t found any solid evidence if it can really reach this speed and what would be the actual technique to reach that speed. Different switch models can be mixed in one stack, but if a 3750v2 is present, the entire stack will not use StackWise Plus anymore.

Stackwise commands
There are some useful commands for a stack, especially for the initial setup. ‘show switch’ is the first and most useful: it shows which switches are present in the stack and their priority. ‘show version’ shows the IOS version of each switch. Best practice is to keep them the same. Also, next to the usual ‘show flash:’ you can also use ‘show flashx:’, where x is the number of the switch in the stack to show an individual flash. The running config is stored on each flash, but the flash by itself does not synchronize and thus may differ.

For configuration: ‘switch x priority y‘ sets the switch’s priority for the master election, with y a number between 1 and 15, where 15 is the highest priority. ‘switch x renumber y’ changes switch numbers, so you can make the numbering the same as the physical layout (e.g. the top switch is number 1, the one below that 2, and so on). Unfortunately, both these commands require a reboot of the stack to apply the changes. Best practice would be to do this during the initial setup so you don’t have to fiddle around later.

Stackwise in practice
I have to admit that Stackwise is less impressive in the real world than on paper. Port-channel over multiple switches works, the backplane has plenty of bandwidth, management is easy through one IP, that’s true. But the design of Stackwise is made so the loss of one switch in the stack or a stack cable will not bring down the entire stack or port-channels. But the stack-cables are of good quality and I’ve never heard of one failing so far, and while switches do go down sometimes due to power failures, it’s not the only cause. If the stack is down on his knees due to a bug, a bridging loop somewhere in the network, or anything else that makes it top to 100% CPU, it tends to react bad, flooding the backplane at times, through the stacking cables, making a loop with itself. While this does not cause any trouble in a working network, it does amplify existing issues. The chassis-based switches like 4000 and 6500 series, while vastly more expensive, do not have this behaviour. And similar to the chassis-based switches, upgrading the IOS does require the entire stack to reboot. Upgrading the IOS can also be a pain because the stack needs to have enough free space on all flash or upgrading does not work, so you have to check (and clean) this manually on each flash before proceeding. Even then, the actual upgrade takes significantly longer in a stack, with the CPU spiking to 100% during IOS install.

StackPower in theory
StackPower is relatively new. It’s a feature on the 3560X and 3750X series of switches.

By connecting them together with special StackPower cables, they share their power supplies into one big pool. You can connect up to four switches together this way, and they don’t necessarely have to be connected bij StackWise. Since the 3560X and 3750X series both support PoE+ and redundant, hot-swappable power supplies, it’s a serious advantage in the design. Should the power supply of one of these switches fail, then power can be drawn from other power supplies in the stack to keep the switch running, assuming there’s enough power left to do so. The switches support two power supplies with up to 1100W per power supply, so a fully loaded stack of four switches can power up to 8800W. If the switches are deployed in an office environment depending heavily on PoE for IP Phones, access points, and so on, then the added power redundancy is welcome. It also requires less backup power supplies to be present: one extra power supply per four switches, instead of per switch, still guarantees redundancy.

StackPower commands
It’s relatively easy to configure: plug in the cables, create a common group name, activate the ports. Creating a group name is done with ‘stack-power stack name‘. The name has to match of course. After entering the command, the CLI allows you to set the power sharing mode: ‘mode power-shared strict’ or ‘mode power-shared non-strict’. Non-strict allows the switches to exceed the power budget if deemed necessary in case of failures, strict does not do this. I personally prefer strict to play safe.
Activating the StackPower ports is not that clearly explained by Cisco. First type ‘stack-power switch x‘, and in this configuration mode, type ‘no standalone’.
Because these are PoE switches, if there’s not enough power left to power every switch in the stack, the switches will stop giving power to PoE devices up to the point where there’s enough power to keep all switches running. It’s possible to influence the decisions made in this process: in interface configuration mode, ‘power inline port priority high’ will assign a higher priority to a PoE port, which will make sure PoE is not taken away from this port until there are no other PoE ports left.

Troubleshooting commands are ‘show env power all’, ‘show stack-power’, and ‘show power inline’. These contain all info that is needed.

StackPower in practice
There’s only one downside in practice: the switches need an IP Base or IP Services IOS licence. LAN Base does not support it. I’ve only had limited experience with it so far, but it seems to do what it promises: failure of a power supply does not cause any disruptions.