Port-aggregation: lifting the confusion.

I’m seeing a lot of people discussion port-aggregation on fora lately, and it seems there are a lot of misconceptions about this. In this article, I’m going to review some mechanics of port-aggregation.

First a quick review of port-aggregation: it is the binding of multiple physical links between two devices to one single logical link, for increased bandwidth and redundancy. In Cisco terminology it’s often called a Port Channel or EtherChannel.

Port-aggregation is a layer two thing and is configured on switches, not routers. People have asked me what’s the difference with HRSP/VRRP, as they both provide redundancy. If you need to ask that, it’s best that you start over and learn the OSI model again (yes, small rant, but really…). Since it’s layer two, it has nothing to do with IP’s.

The links have to be of the same speed, duplex mode and media type, but they do not have to be of the same length (that would be impossible in most real-world situations). If one of the links falls back from 1 Gbps to 100 Mbps or from full duplex to half duplex, it is excluded from the channel. This may happen if a copper pair gets damaged, as 1 Gbps uses all four copper pairs in a wire, and 100 Mbps and half duplex use less pairs.

There are three types of port-aggregation: static, LACP or PAgP. PAgP is Cisco proprietary, LACP is not. Both send control frames on each physical link and will try to negotiate a port-channel. This is recommended because if you do a static configuration but configure the wrong port, a loop may occur.
A static channel can be useful if you need to set up port-aggregation to a device that does not support LACP. Usually this is a server with multiple NICs, like a VMWare ESX. The standard vSwitch supports port-aggregation but no LACP.

Because the links form one logical channel, it becomes one link for STP. If you have the choice, port-aggregation is always better than two separate links. There is not a single argument that will convince me otherwise. Port-aggregation failover occurs in less than a second, perhaps millisecond range, which STP can’t compete with even when fine tuned. It also makes no sense to have two or more links between two switches and just use one of them.
There is no notable difference in latency between one physical link or a port-channel, so it’s not a factor to consider in any design.

I often see a discussion that ‘the total bandwidth of two links is 400 Mbps because it’s full duplex’. In total, yes, but it’s still just 200 Mbps in one direction. Since it’s not said that a single 100 Mbps link ‘is 200 Mbps because it’s full duplex’, it’s better not to listen to the marketing talk and use the single numbers. A modern network has no hubs – You can assume everything is full duplex, and keeps things simple when calculating bandwidth.

The frame still has to travel through one of the physical links. The physical link is chosen based on a hashing algorithm: source and/or destination MAC address, source and/or destination IP address, and on a 6500 chassis source or destination port. This means that a ‘conversation’ always travels over the same physical line: if two computers do a data transfer, they use the same MAC, IP and port during the entire transfer, on both sides. So a single transfer can never take more bandwidth than one single physical link: over an 8 Gbps port-channel or eight 1 Gbps links, the maximum speed of a file transfer still is 1 Gbps. Port-aggregation is designed to be used between multiple hosts.
This also means that the theoretical maximum bandwidth is rarely reached, as one of the links will usually reach 100% before the others do. In case of just a few devices combined with a bad hashing algorithm, throughput may still be low as all transfers go through one physical link.

Conversations going through one link does come with an advantage. Because port-aggregation links can be of a different length, sending frames of the same conversation over different links could cause some jitter, which is bad for things like voice and streaming. A 6500 chassis allows something called ‘adaptive hashing’ which would divide frames more evenly over the links.

And the last interesting fact about port-aggregation: it’s only possible between two devices, but the definition of ‘device’ can be stretched here: a stack of switches (e.g. Cisco 3750) can act as one logical switch, with one management IP and each switch appearing as another module. E.g. a link on the first switch, port 12 (F1/0/12) and a link on the third switch, port 20 (F3/0/20) can be used together in a port-channel. Another example is port-aggregation starting from two different FEX’es, which appear as a module in a parent Cisco Nexus (see my introduction to the Nexus series for more information). Two 6500 chassis can also be used together as a Virtual Switch System (VSS), which is one logical switch.
The only notable exception with Cisco is the Nexus series, where two Nexus 5000 or 7000 can form a port-channel with a third device, with a management link between them to synchronize everything.