If you’ve had little or no real-world experience inside a data center or large switched infrastructure, the Cisco Nexus series of switches is something you probably haven’t encountered so far. Yet, they are rather different from ‘standard’ Cisco Catalyst switches like the 3560/2960/3750 series switches which are most commonly used these days in certification training and most business environments. Since I’ve been able to get my hands on them, I’ll share my experiences to the reader. I’ll be focusing on the 5000 and 2000 series, as these show a clear design difference with the Catalyst series.
A Nexus 2000 is also called a fabric extender, or FEX. The idea is that they extend the switching fabric of a Nexus 5000 or 7000 (the 7000 is a chassis). A FEX has no management interface, but instead has to be connected to a Nexus 5000 or 7000, after which it becomes a logical part of that parent switch. A 32-port Nexus 5000 with ten 48-port Nexus 2000 attached will list a whopping 512 ports under ‘show ip interface brief’, not counting any VLAN interfaces. All interfaces will show as ‘ethernet’, no matter their link speed, so no guessing ‘was it f0/1 or g0/1’ here.
Connection from FEX to parent switch is done via a SFP module with fiber, or a Cisco twinax cable, which is an ethernet-like copper cable with the SFP already attached to it on both sides. Depending on the FEX model, there are two or four SFP uplinks possible, just like most Catalyst switches.
The 5000 series has 32 to 96 1/10 Gbps SFP ports. These ports cannot negotiate any lower than 1Gbps, so 10 or 100 Mbps is not an option. As the parent switch, it is supposed to provide uplinks to other parts of the network, or uplinks to the FEX’s, so high bandwidth is needed. The actual links to the servers are meant to be patched on the FEX’s, which have 24 to 48 100/1000 Mbps ports. 10 Mbps is not possible here. (Frankly, who still uses that?)
An interesting feature is that you can use two 5000 or 7000 together as one logical switch when setting up port aggregation, as long as they have a direct connection between themselves for control. So by using an uplink to another switch or FEX on one Nexus, and using a second uplink on the second Nexus, you can create an Etherchannel, without any of the links getting blocked by STP and without causing a loop. The link between the two Nexus switches will keep information synchronized. This is called a virtual Port Channel or vPC.
Also, they don’t run the classic Cisco IOS, but use NX-OS. While this originally evolved from a different line of operating software, the basic commands are the same as in IOS. Some things are somewhat different, e.g. SPAN or port mirroring requires additional commands. Just for reminder, a SPAN port is configured on a Catalyst switch like this:
switch(config)#monitor session 1 source interface g0/4
switch(config)#monitor session 1 destination interface g0/5
The above will copy all traffic from interface g0/4 to g0/5. If you connect a capturing device on port g0/5 (e.g. a computer with Wireshark running), you can see the traffic. A Nexus works different:
switch(config)#monitor session 1 source interface e111/1/20
switch(config)#monitor session 1 destination interface e1/5
switch(config)#interface e1/5
switch(config-if)#switchport monitor
switch(config-if)#exit
switch(config)#no monitor session 1 shut
By explicitly configuring the switchport as a monitoring interface, there’s less confusion: in the Catalyst series the monitoring switchport can have an entirely different configuration, but it won’t take effect as soon as it becomes a SPAN destination. The monitor session doesn’t start by default, hence the last command. Since you’re working in a multiple gigabits environment, this is an understandable choice.
Using NX-OS has another reason, of course. The Nexus series can run FCoE natively. For more information, read this first. By combining this with servers that have converged network adapters (CNAs) and connecting the Nexus to a SAN, it’s possible to run both storage and IP-based communication through the same physical network.
These are the main reasons Cisco is having success with these lines of switches: they’re very redundant (vPC, dual power supplies, dual fans,…), they provide both LAN and SAN functionality, and have high throughput rates (1/10 Gbps, sub-millisecond switching from server through FEX to parent switch). They are mostly used in an environment that needs large layer 2 domains, like data centers. I’ve also heard of implementations for an access layer design towards many end users, which would work and provide great redundancy, but since these switches weren’t designed with that in mind, they lack PoE capabilities often needed for IP Phones and access points.
Reggle 