IT management/strategy

Unix tip: Monitoring network switches

February 4, 2009, 02:37 PM — 

Helping a friend with a problem on one of his Cisco switches, I noticed that all of his switches were logging to one of his admin servers and that the log they were all writing to was not only getting very large but that it contained several years worth of messages, including a large number of errors. While it's certainly a good idea to centralize log files since this strategy means you will have a way to check on the status of all of your switches at once, you have to remember to periodically check the centralized log for evidence of problems. Otherwise, you lose all the benefit it could be providing in monitoring the health of your LAN.

Many of the errors in the lengthy log file were surprisingly vague. They reported only that various ports on some of the switches were "experiencing errors" and failed to mention the nature of the errors. Here's an example:

Feb 02 11:10:12 switch11-2 241: 000238: Feb 02 11:10:11 UTC: %LINK-4-ERROR:
FastEthernet0/7 is experiencing errors

When we checked the switches and the systems attached to the ports for which these errors were generated, we found that there was a mismatch between the duplex settings. Either the switch port was set to full duplex and the network adaptor on the server was set to half duplex or vice versa.

In case, these terms aren't in the lingo you sling in your day to day life, let's quickly review what they mean. If a network interface is operating in half duplex, it means that systems can communicate through that interface in either direction, but only one direction at a time. If that same network interface is operating in full duplex, it means the systems can communicate in both directions at the same time.

While the systems for which these complaints were being logged were all still operational, network performance is reduced when duplex settings are not the same on the switch port and the connected system. In addition, intermittent connectivity can occur. So, we decided to systematically attack the problems. First, we generated a list of the systems showing errors by extracting and summarizing the errors in the log file like this:

boson# grep "is experiencing errors" cisco.log | awk '{print $4,$12}' | uniq -c
   635 switch11-2 FastEthernet0/16
   117 switch4-10 FastEthernet0/7
   ...

We then compared the settings on each switch port with the settings on the network interface on the Solaris servers. To determine whether a network interface is running in half or full duplex, we used the ndd command. For example, you can tell in the output below that the network interface on boson is running in full duplex (fdx) mode.

boson# ndd -get /dev/dmfe1 adv_100fdx_cap
1
boson# ndd -get /dev/dmfe1 adv_100hdx_cap
0

We then compared the ndd output with the output of the "show interfaces" command on the Cisco switches.

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

I like it!
Comments

Linux ethtool

On a linux system, as root you can run "ethtool eth0", and it will tell you your speed and whether or not you are running in Full or Half duplex mode.
| reply

Nice thought

Welcome to the blog! Enjoy!
Hello everyone,
I'm a newbie to website design and being a this site. I'm stopping by a few forums to pick up tips and get answers from people who know a lot more about this that I do!
My username has to do with cooking, my big hobby. It's related to my website and blog.
How is everyone doing? Would just like to give a quick introduction about myself, I am jack from Arizona, 22 years old and just recently gotten into this and stuff....

seo ppc and it jobs uk
| reply

More info on topic

If you do not have access to the Cisco logs, you can execute "netstat -i" and if Oerrs and Ierrs are not equal to 0 you probable have a problem. Also your network performance will be terrible.

On Solaris 10 you can execute "dladm show-dev" as root to show you the speed and duplex of your network interfaces. I still use "ndd" to set modify the network etherface but you might be able to use "dladm".

I wrote a simple script that uses a combination of "ifconfig -a","netstat -i","netstat -r", "netstat -rav" and "dladm show-dev" to give be all the info I need.
| reply
peer-to-peer

jfruh
Apple syncing patent can't come soon enough

pasmith
New Twitter features borrow from 3rd party clients

Esther Schindler
Open Source Changes the Software Acquisition Process

mikelgan
How to set up continuous podcast play on the new iTunes

David Strom
Five important Windows 7 mobility features

sjvn
Guard your Wi-Fi for your own sake                        

Sandra Henry-Stocker
Grepping on Whole Words

 

Sidekick: The Good News & the Bad News
Either way you look at it Microsoft Data Center management did not follow standards or best practices in this failure. In which case it makes me wonder more about the outsourcing of corporate data much less personal data.
- mburton325

Join the conversation here

The Daily Tip

The Daily TipQuick, practical advice for IT pros. Made fresh daily.

Hot tips:

Want to cash in on your IT savvy? Send your tip to tips@itworld.com. If we post it, we'll send you a $25 Amazon e-gift card.

Newsletters

Subscribe to ITWORLD TODAY and receive the latest IT news and analysis.

I would like to receive offers via email from ITworld partners.
By clicking submit you agree to the terms and conditions outlined in ITworld's privacy policy.
Featured Sponsor

AISO founders envisioned a Web hosting company that was environmentally friendly. While the company employed energy-efficient innovations like solar panels, its infrastructure produced unacceptable power and cooling requirements. Find out how AISO leveraged AMD technology to overcome their challenge in this case study white paper.

In this whitepaper, Scalar explores the opportunity to change the landscape with respect to mission critical databases built around Oracle. Leveraging technologies such as Linux, high-end commodity processing power and Oracle RAC technology to architect, design, build and maintain database infrastructure that delivers maximum availability, reliability and performance at a fraction of traditional cost.

On a typical day, weather.com, the Web site for The Weather Channel in Atlanta, serves up between 15 million and 20 million page views. But in September 2004, when back-to-back hurricanes ransacked Florida, the peak traffic on one day more than tripled: over 70 million page views by more than 7 million unique visitors. Read the full success story now.

Marketplace