Cisco Live 2013: a great experience.

Last week a colleague and I went to Cisco Live London 2013! Overall experience: great. I gained an impressive amount of knowledge in a short time, and writing all that down would generate enough blog posts for a year. We went to many different seminars that really went into the details. Too much to name, but a quick review of the impressive stuff I saw:

• The launch of the Catalyst 3850 switch. I’m not going to read the data sheet, but quick overview: all 3750X functionality (StackWise, StackPower, hardware IPv6, QoS, redundant power and fans), 802.3at PoE+, 32k CAM table possible, integrated Wireless LAN Controller, up to 40 Gbps WLC throughput, and besides the 24 or 48 Gigabit ports up to 4x 10GE is possible in a SFP+ module. Very promising, although most functionality is expected for a next-generation switch, and I have yet to test it all in a production environment.
• A seminar about the Nexus 6000 switch. Designed towards ultra low-latency implementations. While I did think initially that it would be of little use cost/benefits-wise outside of the trading and high computing markets, an article by Greg Ferro made me think about this again. Apparently he was there too in London. Worth the read.
• The Nexus 7000 architecture seminar. This was a bit of a disappointment. Cisco’s flagship data center switch sounds like an elephant: huge, strong and loudly announcing it’s there, but too many features that aren’t ready yet, so watch out what you want to do with it. I noticed plenty of limitations on the current generation that will probably go unnoticed if you’re not using the Nexus to its full potential, but no one buys a Nexus 7000 just for basic switching.
• Energywise fundamentals and deployment by John Parello. An unexpectedly useful technology, that I have working now (more about that in a later blog post). I find that this requires more attention and research.
• FCoE practical lab: a complete overview of the technology, much more in-dept than my own brief review. Again an interesting technology that will prove useful when mastered. For practical implementation it requires a lot of planning and design in advance though.
• Ultra Low Latency Data Center design: I already mentioned some details on latency in my article about fibers. Though not that important for me in practice, I learned some key points about latency.
• Multicast troubleshooting: Luc De Ghein did a great explanation there, giving me insight into multicast inner workings.
• Meet The Engineer meeting with Lars, an engineer sharing the same native language as I did who is a Catalyst 6500 switch expert. He provided me good insights with hardware and software ACLs and Control Plane Policing.
• CCIE Troubleshooting lab: me and my colleague actually managed to solve part of it… But the 45 minutes time was just too short. A nice taste of things that may come for me one day.

For a network engineer this was a really great event, much to see, to learn, to experience. Probably a once-in-a-lifetime chance for me, and I don’t regret it.

Dynamic Trunk Protocol: Wireshark research.

Time to review something that has been very unclear to me from the beginning: Dynamic Trunking Protocol. First off, it’s a Cisco-only protocol, and let’s be honest, it has no real world uses except acting as a security risk. Second: most literature about it is in fact quite unclear, except the CCIE OCG so far, but even then I’m left with question marks.

So let’s go through the theory first, as this is usually the best way to start:

• DTP sends special frames out of every switchport by default, which can be used to negotiate a trunk link, and negotiate the encapsulation (802.1q or Cisco-proprietary ISL).
• The frames can either be ‘auto’, ‘desirable’, or ‘on’. While ‘desirable’ and ‘on’ actively try to negotiate a trunk, ‘auto’ only negotiates one when the other end is set to trunking.
• The ‘switchport nonegotiate’ command prevents DTP frames from being sent.

Now, let’s see how it goes in practice.

Commands without effect
It’s never mentioned explicitly in any literature I’ve encountered so far, so I tested it just to be sure. The commands ‘switchport access vlan number‘ and ‘switchport trunk allowed vlan numbers‘ do not have any effect on DTP at all. Should the port become an access port, then the ‘switchport access’ command is used to define the VLAN, and the trunk allowed list is ignored. Should the port become a trunk, the trunk allowed list is used and the access VLAN is ignored. Same for ‘switchport trunk native vlan number‘, which is only used should the port become a trunk.

Default configuration
The default configuration of a Cisco switch is to send out DTP auto frames. This means no trunk link will be formed unless configured to do so. And here’s the security risk: if you leave a port in it’s default configuration, someone sending a DTP desirable frame can form a trunk and gain access to all VLANs. In case you’re thinking ‘Oh, this is highly theoretical’: it took me less than 30 minutes to find, download and install Yersinia on a Linux VM and send DTP frames to my home lab switch.

Commands with effect, and their changes
So what actually influences DTP? Two commands, it seems: ‘switchport mode’, and ‘switchport nonegotiate’. I’m listing my findings below, after trying all possible combinations on a switch:

• Default configuration: DTP sends two frames every 30 seconds (a timer that can’t be changed): one untagged frame, one ISL encapsulated frame. Both are DTP ‘auto’ mode (status code 0x03). Since ISL has no concept of native VLAN and some (older) switches only support ISL, sending the second ISL frame makes sense for compatibility reasons. DTP also sends a DTP Type, which is a HEX code listing the supported encapsulation methods.
• ‘switchport mode access’: no DTP frames are sent. At all. The ‘switchport nonegotiate’ does not make any difference.
• ‘switchport mode trunk’: DTP frames are sent in mode ‘on’, with a status code of 0x81. Since you have to choose either ISL or 802.1q as an encapsulation method before you can make the port a trunk, DTP sends this parameter too: 0xa5 for 802.1q and 0x42 for ISL. Also, independent of the encapsulation chosen, DTP will again send two frames: one untagged, one ISL encapsulated.
  Note that this port will operate in trunk mode regardless of what the other end decides. The frames are sent to help the other end in negotiation and choosing the right encapsulation. A port with default configuration will convert to a trunk port when receiving these DTP frames.
  Using ‘switchport nonegotiate’ here stops the sending of DTP frames. Should this port connect to a port with default configuration now, it will still be a trunk, but the other and will stay an access port.
• ‘switchport mode dynamic auto’: this is the default configuration and this line will not even show in the running config.
• ‘switchport mode dynamic desirable’: again like the default configuration, the only difference being status code 0x04, and a trunk will actively form when a port running DTP is connected (no matter the other end’s DTP mode).

Another unexpected command which has effect is the ‘vtp domain name‘: I’ve researched that before.

Practical implications
So what’s best in practice to counter any unexpected behavior, and security risks?
In case of an access port, ‘switchport mode access’ is effective enough as it shuts down DTP on a port. ‘switchport nonegotiate’ is a redundant command, so it doesn’t matter whether it’s applied or not. Just defining ‘switchport access vlan number‘ without the matching ‘switchport mode access’ is a security risk.
In case of a trunk port, I would still recommend disabling DTP, so a static configuration with ‘switchport nonegotiate’ becomes mandatory.
And what if DTP somehow is required to run, can the risk be minimized? (This is theory, if someone requires DTP, you… try to stay calm.) Yes: ‘switchport trunk allowed vlan numbers‘ can be configured to only allow access to the default access VLAN, or even set to ‘none’. This command doesn’t become effective until a trunk is negotiated.
Last, the Nexus series doesn’t run DTP at all, which I personally regard as a huge plus.

So that’s my in-dept research. Probably nothing I will really need in real life, but good to know for the security consequences, as well as some more knowledge for the next certification.

BPDU Guard and BPDU Filter: the details.

I’m noticing a shift in focus in my articles: from theory to a more practical approach. This one, hopefully, is a mix of both: BPDU Guard and BPDU Filter.

Let’s start with the theory first: for those who don’t know, BPDU Guard is a feature on Cisco switches which causes a switchport to shut down as soon as it receives a spanning-tree frame. BPDU Filter doesn’t shut down the port, but instead filters out the BPDU, as if it was never received on that port. Both features can be configured globally on the switch, or on a per-port basis. Now to the practical side of things:

BPDU Guard
A port with BPDU Guard will still send out BPDU frames itself. This comes in handy, as two access ports with PortFast and BPDU Guard connected to each other in a Campus LAN will detect the BPDUs and shut down the port. This is a good practice, as you don’t always control what end users connect to the network: their own switches, hubs, computers with network cards placed in bridging mode… All of these aren’t a real problem until somehow a second connection is made. Using the command ‘spanning-tree portfast bpduguard default’, each PortFast port will automatically have it configured. You can even configure ‘errdisable recovery cause bpduguard’ and the switch will recover automatically from these incidents, no need for manual intervention.
But… There are downsides. First of all, the port isn’t shut down until an actual BPDU frame is received: since spanning-tree does this every two seconds by default, a broadcast frame can still loop for two seconds before the port is disabled. That might not be impressive, but it’s sometimes enough to briefly send a spike through the network. Also, while this does stop most loops made through IP Phones, I’ve heard people say some IP Phones filter out BPDU frames. And while the errdisable autorecovery saves management overhead, the loop will briefly reappear as long as the problematic cable isn’t removed.

BPDU Filter
BPDU Filter on a port filters out any received BPDU frames, but at the same time, it stops sending BPDUs as well. It effectively stops spanning-tree from operating on that port, so this port will stay enabled no matter what happens to the spanning-tree topology. Very tricky, as it’s always possible someone misplaces a cable (and we’re all human, so that will happen eventually).
Also… While ‘spanning-tree portfast bpdufilter default’ seems to enable it globally on the switch, it works different: it doesn’t send out BPDU frames on PortFast ports, but once a BPDU frame is received, the port loses PortFast status and starts working like a normal switchport, sending out BPDUs again.
Personally, I strongly dislike experimenting with spanning-tree in such a way, but in a rare occasion, it might be the only option. That rare occasion usually involves a Nexus 2000 Fabric Extender: the access ports are hardcoded with BPDU Guard enabled, so the only way to connect a switch to it is if that switch does not send out any BPDUs. A risky setup.

Any comments? Feel free to share the bridging loop horror stories!

Introduction to Cisco Nexus Fabric Extenders.

If you’ve had little or no real-world experience inside a data center or large switched infrastructure, the Cisco Nexus series of switches is something you probably haven’t encountered so far. Yet, they are rather different from ‘standard’ Cisco Catalyst switches like the 3560/2960/3750 series switches which are most commonly used these days in certification training and most business environments. Since I’ve been able to get my hands on them, I’ll share my experiences to the reader. I’ll be focusing on the 5000 and 2000 series, as these show a clear design difference with the Catalyst series.

A Nexus 2000 is also called a fabric extender, or FEX. The idea is that they extend the switching fabric of a Nexus 5000 or 7000 (the 7000 is a chassis). A FEX has no management interface, but instead has to be connected to a Nexus 5000 or 7000, after which it becomes a logical part of that parent switch. A 32-port Nexus 5000 with ten 48-port Nexus 2000 attached will list a whopping 512 ports under ‘show ip interface brief’, not counting any VLAN interfaces. All interfaces will show as ‘ethernet’, no matter their link speed, so no guessing ‘was it f0/1 or g0/1’ here.

Connection from FEX to parent switch is done via a SFP module with fiber, or a Cisco twinax cable, which is an ethernet-like copper cable with the SFP already attached to it on both sides. Depending on the FEX model, there are two or four SFP uplinks possible, just like most Catalyst switches.

The 5000 series has 32 to 96 1/10 Gbps SFP ports. These ports cannot negotiate any lower than 1Gbps, so 10 or 100 Mbps is not an option. As the parent switch, it is supposed to provide uplinks to other parts of the network, or uplinks to the FEX’s, so high bandwidth is needed. The actual links to the servers are meant to be patched on the FEX’s, which have 24 to 48 100/1000 Mbps ports. 10 Mbps is not possible here. (Frankly, who still uses that?)

An interesting feature is that you can use two 5000 or 7000 together as one logical switch when setting up port aggregation, as long as they have a direct connection between themselves for control. So by using an uplink to another switch or FEX on one Nexus, and using a second uplink on the second Nexus, you can create an Etherchannel, without any of the links getting blocked by STP and without causing a loop. The link between the two Nexus switches will keep information synchronized. This is called a virtual Port Channel or vPC.

Also, they don’t run the classic Cisco IOS, but use NX-OS. While this originally evolved from a different line of operating software, the basic commands are the same as in IOS. Some things are somewhat different, e.g. SPAN or port mirroring requires additional commands. Just for reminder, a SPAN port is configured on a Catalyst switch like this:

switch(config)#monitor session 1 source interface g0/4
switch(config)#monitor session 1 destination interface g0/5

The above will copy all traffic from interface g0/4 to g0/5. If you connect a capturing device on port g0/5 (e.g. a computer with Wireshark running), you can see the traffic. A Nexus works different:

switch(config)#monitor session 1 source interface e111/1/20
switch(config)#monitor session 1 destination interface e1/5
switch(config)#interface e1/5
switch(config-if)#switchport monitor
switch(config-if)#exit
switch(config)#no monitor session 1 shut

By explicitly configuring the switchport as a monitoring interface, there’s less confusion: in the Catalyst series the monitoring switchport can have an entirely different configuration, but it won’t take effect as soon as it becomes a SPAN destination. The monitor session doesn’t start by default, hence the last command. Since you’re working in a multiple gigabits environment, this is an understandable choice.

Using NX-OS has another reason, of course. The Nexus series can run FCoE natively. For more information, read this first. By combining this with servers that have converged network adapters (CNAs) and connecting the Nexus to a SAN, it’s possible to run both storage and IP-based communication through the same physical network.

These are the main reasons Cisco is having success with these lines of switches: they’re very redundant (vPC, dual power supplies, dual fans,…), they provide both LAN and SAN functionality, and have high throughput rates (1/10 Gbps, sub-millisecond switching from server through FEX to parent switch). They are mostly used in an environment that needs large layer 2 domains, like data centers. I’ve also heard of implementations for an access layer design towards many end users, which would work and provide great redundancy, but since these switches weren’t designed with that in mind, they lack PoE capabilities often needed for IP Phones and access points.

Virtual switching – A short introduction.

Virtual switching plays an important role in the data center, so I’m going to give a brief overview of the different products. What is virtual switching? Well, a physical server these days usually has a hypervisor as operating system, which has only one function: virtualizing other operating systems to virtual machines that are running on top of the hypervisor. These virtual machines can be Windows, Linux, Solaris, or even other operating systems. These virtual machines need network connectivity. For that, they share one or more physical network interface cards on the server, commonly called a pNIC. To regulate this network traffic, a virtual switch, called a vSwitch, runs in software on the hypervisor and connects these pNICs with the virtual network interface cards of the virtual machines, called vNICs. So it looks like this:

The blue parts are done in software, only the last part, the pNIC, is physical.

There are three big players in the hypervisor market: Citrix with XenServer, Microsoft with Hyper-V and VMware with ESXi or vSphere. Each has their own implementation of a virtual switch.
Apart from that, Cisco has a Nexus 1000 virtual switch.

Citrix Xenserver
I have no experience with XenServer and so far I’ve found litte information on it. A virtual switch that can be used is Open vSwitch, an open source product which runs on Xen and Virtualbox. I’m not sure if this is the only virtual switch that XenServer supports. Open vSwitch supports a variety of features you would expect from a switch: trunking, 802.1q VLAN tags, link aggregation (LACP), tunneling protocols, SwitchPort ANalyser (SPAN), IPv6, basic QoS. I could not find anything in regard to Spanning Tree Protocol support, so I’m uncertain what will happen if a loop is created to a server with multiple pNICs and no link aggregation configured.

Microsoft’s Hyper-V
Again, I have little real world experience with Hyper-V, and details are not clear, but the virtual switch supports the mandatory 802.1q VLAN tags and trunking. Advanced spanning-tree support is missing as far as I can tell, you can’t manipulate it. I’ve found no information on link aggregation support. It’s a very simple switch compared to the other products. There’s one advantage though: you can run the Routing and Remote Access role on the Windows Server and do layer 3 routing for the VMs, which offers some possibilities for NAT and separate subnets without the need of a separate router. It’s a shame Microsoft decided to no longer support OSPF on their Windows Server 2008, as this might have been a great addition to it, making a vRouter possible. RIPv2 should still work.

VMware’s ESXi and vSphere
The vSwitch developed by VMware is, in my opinion, very good for basic deployment. It supports 802.1q VLAN tags and trunking. It does not support spanning-tree but incoming spanning-tree frames are discarded instead of forwarded. Any frames entering through the pNICs that have the source MAC of one of the virtual machines are dropped. Broadcasts are sent out through only one pNIC. These mechanisms prevent loops from forming in the network. Link aggregation is present but only a static EtherChannel can be formed, which requires some additional planning. QoS is not supported, and no layer 3 functions either.

Nexus 1000 virtual switch
I’m adding the NX1000V to this list, as it is currently one of the few products on the market that can be used as a vSwitch instead of the default hypervisor vSwitch. Currently there’s only support for vSphere, but Cisco announced that there will be support for the Windows Server 8, too.
The NX1000V is supposed to support anything that’s possible with a physical Nexus switch. So compared to the default vSwitch used, it will add support for LACP, QoS, Private VLANs, access control lists, SNMP, SPAN, and so on.

With the ongoing virtualisation of data centers, virtual switching is an emerging market. For those of you interested in it, it’s worth looking into.