Unix as a Second Language

Unix tip: Monitoring network switches

Helping a friend with a problem on one of his Cisco switches, I noticed that all of his switches were logging to one of his admin servers and that the log they were all writing to was not only getting very large but that it contained several years worth of messages, including a large number of errors. While it's certainly a good idea to centralize log files since this strategy means you will have a way to check on the status of all of your switches at once, you have to remember to periodically check the centralized log for evidence of problems. Otherwise, you lose all the benefit it could be providing in monitoring the health of your LAN.

Many of the errors in the lengthy log file were surprisingly vague. They reported only that various ports on some of the switches were "experiencing errors" and failed to mention the nature of the errors. Here's an example:

Feb 02 11:10:12 switch11-2 241: 000238: Feb 02 11:10:11 UTC: %LINK-4-ERROR:
FastEthernet0/7 is experiencing errors

When we checked the switches and the systems attached to the ports for which these errors were generated, we found that there was a mismatch between the duplex settings. Either the switch port was set to full duplex and the network adaptor on the server was set to half duplex or vice versa.

In case, these terms aren't in the lingo you sling in your day to day life, let's quickly review what they mean. If a network interface is operating in half duplex, it means that systems can communicate through that interface in either direction, but only one direction at a time. If that same network interface is operating in full duplex, it means the systems can communicate in both directions at the same time.

While the systems for which these complaints were being logged were all still operational, network performance is reduced when duplex settings are not the same on the switch port and the connected system. In addition, intermittent connectivity can occur. So, we decided to systematically attack the problems. First, we generated a list of the systems showing errors by extracting and summarizing the errors in the log file like this:

boson# grep "is experiencing errors" cisco.log | awk '{print $4,$12}' | uniq -c
   635 switch11-2 FastEthernet0/16
   117 switch4-10 FastEthernet0/7
   ...

We then compared the settings on each switch port with the settings on the network interface on the Solaris servers. To determine whether a network interface is running in half or full duplex, we used the ndd command. For example, you can tell in the output below that the network interface on boson is running in full duplex (fdx) mode.

boson# ndd -get /dev/dmfe1 adv_100fdx_cap
1
boson# ndd -get /dev/dmfe1 adv_100hdx_cap
0

We then compared the ndd output with the output of the "show interfaces" command on the Cisco switches.

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world