Ethernet grows up -- and out

New methods for carrying native Ethernet frames over metropolitan-area networks (MAN) and WANs will enable applications that simply weren't possible before because they were too expensive to deploy, too bandwidth-constrained or both.

Those applications include everything from off-site backup, storage and disaster recovery all the way up to outsourcing your entire network.

But don't go running out to place a down payment on 10G bit/sec Ethernet just yet. The new version may look a lot like previous incarnations, but under the hood there are big differences in distance, cabling, management and network design requirements.

What's the same

Because the new specification really is "just Ethernet," your investment in Ethernet gear and in the training of your network staff is protected.

The minimum (64 bytes) and maximum (1,518 bytes) frame lengths haven't changed and the Ethernet frame format also remains in place, so a stream of 10G Ethernet frames will look the same as any other type of Ethernet.

More importantly, 10G Ethernet will carry the same traffic as any other type of Ethernet, including PBX traffic, according to Bruce Tolley, manager of emerging technologies for Cisco's enterprise business unit and vice president of the 10 Gigabit Ethernet Alliance (10GEA).

It's also important to understand what's not in the new specification. That means 10G Ethernet has no built-in facility for quality of service (QoS) or other advanced features. However, there's nothing to prevent network managers from using existing QoS features such as Diff-Serv over 10G bit/sec Ethernet.

What's different

The new version introduces numerous changes at the data-link and physical layers, including new interfaces intended specifically for MAN and WAN use.

Additionally, the new specification will run only in full-duplex mode, while every other type of Ethernet allows for half-duplex operation.

Another change that's probably welcome is that 802.3ae does not support autonegotiation, which was intended to be a convenience but in practice has proven to be a major source of connectivity headaches. The elimination of autonegotiation is likely to simplify troubleshooting.

The biggest difference is at the physical layer. Notably, the new standard will include two PHY (physical-layer) types: a LAN PHY that operates at 10G bit/sec and a WAN PHY that runs at 9.58464G bit/sec -- the same payload rate as SONET OC-192.

Seamless integration with SONET MANs and WANs was a key design goal for the IEEE task force. Even though pure Ethernet MANs are coming on strong, SONET is king in the telecom world -- and it's likely to remain so for years to come. Moving Ethernet frames over SONET networks with a minimum of disruption will help extend Ethernet into MAN and WAN usage.

To deal with the rate mismatch between the LAN and WAN PHYs, the IEEE task force defined a pacing mechanism in the media access control layer that adds enough idle signals to the interframe gap to slow the data rate from 10G to 9.6G bit/sec for transmission by the WAN PHY. Then, a physical-layer component, the WAN interface sublayer (WIS), handles SONET framing, scrambling and error detection.

The WIS is not a complete SONET-compliant interface, but rather a lightweight SONET-like framer aimed at ensuring rate compatibility.

The result is that Ethernet switches equipped with SONET interfaces (or SONET elements such as add/drop multiplexers equipped with 10G Ethernet interfaces) can move Ethernet traffic across SONET MANs and WANs at OC-192 rates. And SONET management systems will be able to identify and monitor Ethernet traffic encoded with SONET framing, an important requirement for service providers.

Distance

Of course, for any PHY to be credibly called a MAN or WAN interface, it has to reach farther than the 5-kilometer limit currently defined for Gigabit Ethernet. To deal with the extended-distance requirements, the IEEE task force defined four new physical-media-dependent sublayers: three serial Physical Medium Dependents operating at 850, 1,310 and 1,550 nanometers, and a wide wavelength division multiplexing (WWDM, sometimes called coarse WDM) PMD operating at 1,310 nanometers. The 1,550-nanometer interface will operate at distances of up to almost 40 kilometers.

Serial PMDs do just what the name implies: They send signals in series, one after another.

With any type of WDM, a transmitting interface sends light over multiple wavelengths in parallel, and a receiving interface reassembles the wavelengths into a whole.

With all the new variations in PHY types, it can be confusing to keep track of all the possible interfaces. The IEEE task force settled on a common nomenclature to make sense of all the new PHY flavors.

The new PMDs are a major departure from earlier versions of Ethernet, in variety and in type. Fast Ethernet borrowed its physical layer encoding from FDDI. Gigabit Ethernet borrowed its encoding from Fibre Channel. In contrast, the 10 Gigabit Ethernet PMDs are all new.

Line encoding is, quite literally, a low-level function that will only affect network managers concerned with troubleshooting physical layer errors. Still, it's incumbent on network managers to verify that cable test gear natively supports 10G Ethernet encoding methods or can be upgraded.

Management issues

If looking after low-level diagnostics is only a minor issue, managing 10G Ethernet's sheer data volume will be a major concern indeed. Network management systems such as element managers, Remote Monitoring (RMON) probes and protocol analyzers already have enough trouble keeping up with current data levels. For these types of devices, tracking 10 times more traffic will be like drinking from a fire hose.

To keep pace with the volume of 10G Ethernet traffic, 64-bit counters will be a must for management and monitoring systems. The 32-bit counters in many currently deployed tools simply can't count high enough. Once a 32-bit counter reaches its limit, it will simply "wrap," or revert to 0 and begin counting again -- and the counts it produces will be meaningless.

Consider a few simple statistics: Given a flow of 256-byte packets (which is roughly the mean packet length for Internet traffic) running at line rate, it will take just 3.7 seconds to wrap a 32-bit byte counter and less than 16 minutes to wrap a packet counter. Clearly, devices equipped with 32-bit counters will be woefully inadequate for long-term monitoring or capacity planning.

However, it may be possible to get more life out of existing tools equipped with 32-bit counters, as long as they support sampling.

Sampling is best-suited for medium- and long-term monitoring. For example, an RMON probe (or RMON agent embedded in a switch) might use sampling to report on port statistics. However, sampling is not appropriate for short-term measurements such as those a protocol analyzer might take. In this case, the ability to monitor and/or capture all traffic in real time is essential.

Cabling

Cabling is a major issue in migrating to any new PHY technology. Network architects will need to determine whether existing cable plants are capable of carrying 10G Ethernet traffic and assess distance limits imposed by the new specification.

Copper is out. Unlike Gigabit Ethernet, which includes a 1000Base-T specification for copper cabling, 10G Ethernet will run only over fiber. This isn't expected to be a showstopper because fiber already represents the majority of Gigabit Ethernet interfaces sold.

But which type of fiber to use?

• On the LAN, multimode is the most likely choice.
• For existing FDDI and Gigabit Ethernet installations, 50- or 62.5-micron multimode is the most commonly installed type of fiber.
• Of the four new PMDs, the 850- and 1,310-nanometer versions will support existing multimode cable plants.
• Interfaces with 850-nanometer lasers are intended for use in very short reach applications, such as links between switches or supercomputers.
• For general-purpose campus connectivity, interfaces with 1,310-nanometer lasers, which light up fiber cabling for up to 300 meters, are likely to be the norm.
• For distances above 300 meters, single-mode fiber will be a must. Interfaces equipped with 1,310-nanometer lasers and single-mode fiber will support distances of up to 10 kilometers (which is twice the limit currently supported in Gigabit Ethernet).
• And if 10 kilometers isn't long enough, interfaces equipped with 1,510-nanometer lasers will reach distances of up to 40 kilometers. It's the 1,510-nanometer flavor of 10G Ethernet that's expected to drive Ethernet growth in the MAN and WAN.

LAN applications

In LAN and campus settings, "the first thing people are going to do with 10G Ethernet is bandwidth aggregation," says Mark Fishburn, vice president of marketing for Spirent Communications and marketing director for the 10GEA. For sites where Gigabit Ethernet doesn't furnish enough bandwidth, 10G Ethernet is an ideal replacement.

A typical candidate for migration to 10G Ethernet might be a switched backbone using Gigabit Ethernet today. Here, workgroup switches with 10M or 100M bit/sec interfaces might employ one or more Gigabit Ethernet links to connect to a backbone device.

The problem here is that the edges can easily overload the backbone. Many workgroup switches offer as many as 48 100M bit/sec ports, which in the aggregate represents far more bandwidth than a single Gigabit Ethernet link can handle.

Link aggregation (also known as trunking or inverse multiplexing) offers more bandwidth, but it's only a partial remedy.

True, switches that support link aggregation can combine up to eight physical ports to create a bigger virtual pipe. But recent tests of link aggregation suggest this feature can have a severe impact on performance, especially when features such as failover or quality-of-service enforcement are enabled.

Even if a device were to support link aggregation with no performance degradation, the technique still would require "burning" multiple interfaces to work. That process is expensive in terms of the capital cost of the interfaces, real estate they consume (especially in cramped wiring closets), cost of buying and pulling multiple runs of fiber cabling, and management resources (human and machine) required to monitor the network.

Removing the bandwidth bottleneck with 10G Ethernet is easy. All that's required to migrate is simply swapping interfaces to achieve a tenfold bandwidth boost. One caveat: Make sure switch or router backplanes have the capacity to accommodate a tenfold traffic increase.

Another application in which 10G Ethernet can ease congestion is in aggregating traffic to and from server farms.

Cisco's Tolley says he expects clusters of computers to be linked using 10G Ethernet, either as an interswitch connection or as a link to storage-area networks.

While it's unlikely any single server will fill a 10G bit/sec pipe any time soon, today there are servers that can move traffic in the 1G bit/sec range. Obviously, a switch/router that tops out at just 1G bit/sec won't be much help in handling traffic from multiple servers -- but a switch that provides one or more 10G bit/sec pathways to the server farm may be just the ticket.

MAN, WAN applications

The place where 10G Ethernet is expected to bring the greatest change is in MANs and WANs. The new specification won't just extend the reach of existing networks; it's also expected to drive entirely new applications that aren't possible with today's bandwidth constraints.

"The ability to get a 10G Ethernet connection [from a service provider] makes the idea of outsourcing your IT infrastructure a lot more compelling, because the connection to that provider is no longer a bottleneck," says Marshall Eisenberg, director of marketing at Foundry Networks.

Tony Lee, president of the 10GEA and senior marketing director at Extreme Networks, expects MAN service providers to be among the first to roll out MAN/WAN versions of 10G Ethernet interfaces. There are compelling economic reasons for doing so: 10G Ethernet interfaces will cost far less than SONET versions. Lee expects incumbent carriers "to leverage SONET investment by running new 10G Ethernet services directly over existing infrastructure."

For example, consider a typical MAN setting today (see Figure 2). To connect multiple offices, a financial services firm leases capacity on a service provider's SONET ring.

Ethernet traffic from each location must be converted for SONET transmission by a switch/router equipped with both Ethernet and packet-over-SONET interfaces. Then it's placed directly onto the SONET ring by add-drop multiplexers.

With 10G Ethernet, the picture is considerably simpler. Here, devices can natively transport Ethernet frames from end to end - even using the existing SONET infrastructure because the new specification's WAN PHYs are rate-compatible with SONET.

WAN connections are also possible because the WAN PHY's framing lets Ethernet traffic be placed directly onto long-haul SONET links.

Using the WAN PHY and leasing dark fiber from service providers, companies can enable the technology to support new uses, such as remote hosting, offsite storage and backup, and disaster recovery.

While the WAN PHY makes the most sense for existing MANs and WANs, Extreme's Lee says it makes the most sense to adopt the LAN PHY for new installations. Lee points out that both types -- LAN and WAN -- run over all four PMDs.

Another big advantage of end-to-end Ethernet service is that it "will allow service providers to provision services in minutes or hours," Lee says.