Gigabit Ethernet switch/routers offer blazing performance in basic configurations, but enabling a feature called link aggregation causes some devices' throughput to tank, latency to leap, failover to falter and quality of service to quit.
the major finding of an evaluation of these devices conducted by Network World Global Test Alliance member Network Test. Teaming with test equipment maker Spirent Communications, we pounded high-end switch/routers from Cisco, Extreme Networks and Nortel Networks with traffic representing more than 14,000 hosts - just the kind of loads these boxes might handle at the core of corporate backbone networks.
Foundry Networks and Avaya originally agreed to participate in our test, but failed to show up in the lab. Alcatel and Enterasys refused our invitation.
All these devices adequately move packets through one Gigabit Ethernet interface at a time. Gigabit Ethernet offers a fat pipe, but even that isn't enough for some enterprise networks.
Vendors say the way to burn through these bandwidth bottlenecks is link aggregation or "trunking," a means of combining up to eight interfaces to form a single, larger virtual pipe.
It's a good idea, but our results suggest there's still work to complete. In assessing basic trunking, failover and QoS capabilities, we found that link aggregation introduced as many problems as it solved. In the worst cases, configurations that worked flawlessly with single interfaces wouldn't work at all when we tried to repeat tests using trunking.
That's not to say our results were all bad news. Cisco's Catalyst 6509 handled most link aggregation tests far better than the other boxes. Some of the 6509's results with two trunks defined were better than with one. Our results suggest Cisco's device is up to the task of delivering vast amounts of bandwidth on corporate backbones, even with advanced features such as QoS enabled. While not all the 6509's numbers were perfect, the Cisco device easily earned the Network World Class award in this round of testing.
It wasn't as pretty a picture for the Extreme and Nortel switch/routers, which were plagued by performance gremlins in many trunking tests. The line-rate throughput normally seen in Extreme's devices plunged as low as 10% of line rate in some tests.
Nortel's boxes delivered generally satisfactory results in the tests completed, but couldn't complete other tests because features such as multiple concurrent trunks and Open Shortest Path First (OSPF) failover aren't supported.
First things first
We conducted four sets of tests: baselines, link aggregation, failover and QoS capabilities.
Baseline tests tell us about a device's basic packet-handling capabilities. In these baselines (see how we did it, page 58), we measured throughput, latency, jitter and packet sequencing.
The first measurement was throughput. Even when forwarding 64-byte packets - the most stressful condition - nearly all the test runs were perfect on the products tested with no packet loss when handling line-rate traffic. The exception is the Cisco 6509, which dropped about 6,000 packets at line rate (out of an offered load of nearly three billion packets). We backed off the load to 99.9% of line rate and the 6509 didn't drop any.
While we expected equally rosy results with large packets, that wasn't the case. In the 64-port configuration, Extreme's Black Diamond couldn't forward loads any higher than 94.6% without loss. Nortel's result is missing in action - we saw packet loss at line rate. Cisco's 6509 forwarded large packets at line rate without any loss.
The throughput numbers don't tell the whole story about packet loss. Cisco and Nortel brought in new 16-port Gigabit Ethernet line cards. Both vendors' cards are blocking, which means they can't forward line-rate traffic on all ports without packet loss.
In Cisco's case, the new card is nonblocking on up to 10 ports, while the threshold for Nortel's new card is eight ports. Both vendors achieved good numbers in this test by supplying enough cards so that they didn't need to use every port of every line card.
Both vendors noted that few production networks run at line rate. While this is true, there are other factors that make this argument dubious.
For example, TCP-based applications, which constitute 90% of all IP traffic, inherently try to use all available capacity for any data transfer. Thus, any time TCP connections try to concurrently move data on more than eight or 10 ports, blocking will occur. The network may still appear to be lightly loaded because most network management equipment samples utilization over some period of time.
What sampling techniques miss is that blocking will prevent a group of instantaneous bursts from using the maximum capacity of the wire. Thus, the instantaneous forwarding rates for these devices will be well below line rate.
The waiting game
Delay-related metrics, such as latency and jitter, are just as important for delay-sensitive applications such as voice over IP or videoconferencing. Latency and jitter should be low and constant to avoid choppy application performance.
Fortunately, latency was a complete nonissue in our tests. Even the highest figures - around 100 microseconds for Cisco and Nortel in the 32-port mesh tests - are an order of magnitude below the point where the delay would disrupt any application.
Extreme's Black Diamond earns bragging rights with the lowest single latency number we recorded, just 5.9 microseconds with short packets in our eight-port test. However, Nortel's Passport exhibited less variation in moving from eight to 32 to 64 ports, and it achieved the lowest average latency of 11.8 microseconds with short packets across all tests.
As expected, latency rose for long packets, but not by much. Extreme again bagged the lowest single latency measurement with 27 microseconds in an eight-port configuration. Cisco's 6509 was the most consistent across all three configurations, achieving the lowest average latency of 60.3 microseconds with long packets.
Jitter - the amount of variation in latency - also turned out to be low and consistent, for the most part. Nortel was the leader, posting jitter of 16.7 microseconds or less in all the 64-byte tests, and 101.6 microseconds or less in the tests with 1,518-byte packets.
Curiously, the Cisco and Extreme switch/routers posted higher jitter than average latency in at least one of the tests with 64-byte packets. The most prominent example is Cisco's 6509 with 32 ports, with jitter of 211 microseconds, which was more than double the average latency.
Jitter needs to climb into the millisecond range before having a noticeable impact on application performance. There is one case in which these numbers could be significant - in a large network comprising many switch/routers. Latency and jitter are cumulative.
A network of 100 switch/routers each adding 200 microseconds of delay will experience at least 20 milliseconds of delay. In that range, application performance could easily suffer.
The final event in our baseline tests checked for out-of-sequence packets. A device that reorders packets can be disruptive to connection-oriented protocols such as TCP, which rely on sequence numbers to verify delivery.
Fortunately, none of the switch/routers have a problem in this area, with only one exception, as all 14.5 billion packets forwarded in our baseline tests arrived in the same order. The exception was Extreme's Black Diamond handling 1,518-byte packets in the 32-port configuration. In that test, the Black Diamond reordered 27 of the 148 million packets it handled.
Link aggregation, also known as load sharing, inverse multiplexing and trunking, lets network managers boost bandwidth on switch-to-switch connections by dedicating up to eight interfaces into one "virtual pipe." All the ports in the trunk share a single IP address, simplifying device management. Aggregated links also add redundancy.
Trunking introduces some downsides. As our test results show, aggregating multiple interfaces into a single link will degrade throughput and latency.
The vendors say their devices use complex algorithms to allocate trunking resources, but it seems these algorithms aren't as highly optimized as those that handle packets from single ports.
We saw big differences between the products in our single-trunk tests. First, no vendor achieved line-rate throughput, so trunking introduces at least some performance degradation. Second, Cisco's 6509 produced the paradoxical result of forwarding short packets at a higher rate without loss than long ones. This is counterintuitive because, everything else being equal, long packets should be much easier for the device to handle.
Link aggregation introduces two new variables to how a switch/router operates.
The switch/router is now charged with scheduling packets to get on the trunk and determining which physical interface within the trunk will carry the packets. Some algorithms take longer to make these decisions for long packets than for short ones.
That's what we saw with Cisco's 6509, which achieved throughput near line rate with short packets, but hit just 93.6% of line rate with long packets.
Extreme's Black Diamond got the short-long order right, but that was little consolation because the switch's throughput with short packets was an anemic 36% of line rate. The Black Diamond offered users a choice of three packet-handling algorithms for trunking: round robin, port-based and address-based. If you experiment with different combinations of these, you'll find the best results come when you use the round-robin algorithm for the backplane and the port-based algorithm for the trunk.
Nortel's Passport 8610 achieved respectable throughput with short and long packets, giving it the best overall showing in the single-trunk test.
It was another story altogether for the test with dual trunks.
The Passport 8610 doesn't support this capability, while Cisco and Extreme support multiple concurrent trunks. Cisco's 6509 handled two trunks better than one, with throughput numbers around 99% of line rate for short and long packets. Conversely, Extreme's Black Diamond suffered more with two trunks than with one. In the worst case, its throughput with short packets is just 10%. Even with long packets, throughput was just 62% of line rate.
Trunking also has an impact on latency (see link aggregation baseline latency graphic, page 54). Cisco's delay with short packets tripled over its 64-port baseline result, and it shot up fivefold in the dual-trunk test with long packets.
Extreme's latency for long packets over dual trunks increased even more over the baselines, by 1,200%. These numbers are below the threshold where they'd disrupt application performance, but again, latency is cumulative over multiple hops - put enough trunks between client and server, and performance could suffer.
We also examined how switch/routers handled the addition or loss of a port from an active trunk. Ideally, a change such as this should have little impact.
In practice, two of the three switch/routers showed significant differences in forwarding rates when the trunk configuration changed.
Nortel's Passport 8610 produced the most consistency in this test (see adding and removing ports from a trunk, below). However, Cisco's 6509 and Extreme's Black Diamond forwarded more packets when we added ports and fewer packets when we subtracted ports. This suggests the Cisco and Extreme devices respond faster to trunk configuration changes than the Nortel switch. Which is better, more consistency or higher peaks and lower troughs? You need to make your own trade-offs between predictability and speed.
Our failover test examined how quickly the switch/routers moved traffic onto a secondary link on the loss of a primary circuit.
The two most popular failover mechanisms are the Layer 2 spanning tree protocol and a feature of Layer 3 OSPF called equal cost multipath (ECMP).
Spanning tree is known for its glacial cutover times. Even on Gigabit Ethernet links, it can take 60 seconds or more for traffic to begin flowing over a back-up link. For these tests, we asked vendors to use OSPF ECMP instead.
We ran the failover test in two configurations: Once with a single link between chassis, and again with an eight-port trunk between boxes.
Extreme's Black Diamond posted the best and worst results here (see failover times graphic, above right). With just one Gigabit Ethernet link between the chassis, the Black Diamond cut over to the secondary link in just 0.31 seconds. But the Black Diamond struggled with the aggregated link, registering a cutover time of 2.53 seconds.
Cisco's 6509 produced nearly identical results with one link or eight between the boxes, taking only more than half a second in both cases. Nortel's single-link cutover of 0.51 seconds edged out Cisco's time, but its device doesn't yet support ECMP failover for trunks.
To see how well these devices prioritize traffic when congestion strikes, we set up a test involving three classes of traffic and two chassis connected over a congested link. First, we offered the three classes of traffic in one ratio (1-to-10-to-10 for the high-, medium-, and low-priority flows, respectively) and asked vendors to deliver the classes of traffic in a different ratio (2-to-12-to-7).
We then asked vendors to deliver the relatively small amount of high-priority traffic with no packet loss. Finally, we ran the test a second time, this time with no high-priority traffic. Here we offered equal amounts of medium- and low-priority packets and asked vendors to deliver the traffic in a 5-to-3 ratio.
To achieve the second ratio, the switch/routers had to allocate bandwidth previously devoted to high-priority traffic to the medium-priority flows.
We did this as a check against static configurations that "nail up" fixed bandwidth allocations for each traffic class and prevent other classes from using bandwidth even when it's available.
In addition to preserving some packets and junking others, we expected the switch/routers to mark the different packet headers' IP precedence fields to indicate the relative priorities of each traffic class.
To create congestion, we offered enough traffic to 21 interfaces on each chassis to present a 2-to-1 overload on the link between the boxes. We repeated the QoS tests with a single Gigabit Ethernet link and a seven-port trunk between the two chassis.
Network Associates' Sniffer traces revealed that Nortel's Passport did not mark packets using IP precedence values but used Differentiated Services code points (DSCP), a traffic classification mechanism now being defined in the Internet Engineering Task Force. That's a commendable feature because it's likely that all vendors eventually will use DSCPs for Layer 3 classification. Unfortunately, Nortel says its switch/routers do not support classification using IP precedence levels.
In the test with three traffic classes, Cisco's 6509 came closest to hitting our desired ratio of 2-to-12-to-17 among the different traffic classes. Cisco's ratio was 2.00-to-12.03-to-6.97, Extreme's was 2.00-to-12.28-to-6.72, and Nortel's was 1.99-to-11.88-to-7.13. All three boxes hit the ratio, but Cisco's result is closest by a few hundredths of a point. Cisco and Extreme also delivered the high-priority traffic with no packet loss, and Nortel's switch/routers dropped a paltry three high-priority packets.
Results differed more when we offered only two traffic classes. Here, only the Cisco 6509s achieved the desired 5-to-3 ratio. The Black Diamonds posted a 5-to-2 ratio and the Nortel boxes delivered in a 5-to-4 margin. These errant ratios suggest that when more bandwidth be-comes available, the boxes will supply too little or too much low-priority traffic.
Things got worse when we attempted the same test with a seven-port trunk linking the boxes. Cisco's 6509 again hit its numbers dead-on: 2-to-12-to-7 but Nortel's ratio was 2-to-11-to-3 and Extreme's numbers were a lopsided 2-to-17-to-3. Clearly, the addition of link aggregation in the Extreme and Nortel switch/routers made it much more difficult for these devices to prioritize traffic when congestion struck. Additionally, the Extreme and Nortel gear dropped significant amounts of high-priority traffic: 7% for Extreme and 23% for Nortel.
It was much the same picture in the test with two classes. Cisco again breezed through, hitting the desired 5-to-3 ratio. Extreme and Nortel severely underfed the low-priority traffic delivering packets in a 5-to-1 ratio.
The lesson is that QoS and trunking can be a hazardous combination. Trunking introduces complexity as switch/routers need to do additional queuing on top of the buffering that QoS requires.
None of the devices we tested achieved perfect results across the board, but Cisco's Catalyst 6509 came the closest. This is especially true in the tests involving link aggregation, where the Extreme and Nortel devices couldn't cope in some events. If speed is the requirement, any of these switch/routers will do an adequate job. However, for more advanced configurations such as those involving link aggregation, fast failover and traffic prioritization, our results suggest that Cisco's Catalyst 6509 is far better suited than its rivals.
This story, "The trouble with trunking" was originally published by Network World.