April 22, 2001, 9:09 PM — Q: I sometimes see messages on the console when a disk has an error. How can I automatically look for a disk that is giving soft errors before it fails completely? Are there any tools that will do this for me?
A: Traditionally, the disk device driver reacts to a problem by printing an error via syslog that gets printed on the console and stored in the /var/adm/messages file. While this is useful, it's an inefficient way to report a problem, and it's difficult to ensure that messages aren't lost or ignored. It's also hard to parse the error messages, because you might not know all the possible error conditions that could be reported.
The SyMON 1.x and 2.x products include a log file monitor that watches the /var/adm/messages file and warns the user if it sees certain kinds of errors. A new way to monitor disks was added in Solaris 2.6, and the iostat command was extended with -e and -E options that report error counts. I've now also extended the SE toolkit to look at the same information as part of the disk monitoring rule.
This isn't normally considered a performance question, but disks that are doing multiple retries due to transient errors can cause hard-to-find performance problems -- and the performance of a dead disk is zero!
Monitoring errors with iostat
Since Solaris 2.6, a new kstat data structure has been maintained in the kernel for every disk. Here is an example output from an Ultra 2 with two disks and a CD-ROM. The disk name is given in the normal device form of sd1 unless the -n option is also used to translate the name to c0t1d0s0 form.
The first line gives the total number of soft errors, hard errors, and transport errors; then the device identification string is broken down into vendor, product, firmware revision, and serial number. This string does not always work out properly, as you can see for the CD-ROM, where the serial number appears to be a date. The format used is vendor specific and difficult to parse in a general way. Next the size is given, and a detailed error count breakdown showing the total number of errors since boot is provided. In this case the system has been powered down using the cpr(7) capability. Each time this occurs, the disks are left spun down until they are accessed, giving rise to a No Device error for each power up resume.
% iostat -E sd0 Soft Errors: 0 Hard Errors: 7 Transport Errors: 0 Vendor: SEAGATE Product: ST32550W SUN2.1G Revision: 0418 Serial No: 04406244 Size: 2.13GB <2127708160 bytes> Media Error: 0 Device Not Ready: 0 No Device: 7 Recoverable: 0 Illegal Request: 0 Predictive Failure Analysis: 0 sd1 Soft Errors: 0 Hard Errors: 7 Transport Errors: 0 Vendor: SEAGATE Product: ST32550W SUN2.1G Revision: 0418 Serial No: 04620031 Size: 2.13GB <2127708160 bytes> Media Error: 0 Device Not Ready: 0 No Device: 7 Recoverable: 0 Illegal Request: 0 Predictive Failure Analysis: 0 sd6 Soft Errors: 0 Hard Errors: 1 Transport Errors: 0 Vendor: TOSHIBA Product: XM-5401TASUN4XCD Revision: 1036 Serial No: 04/12/95 Size: 18446744073.71GB <-1 bytes> Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 Illegal Request: 0 Predictive Failure Analysis: 0
Underlying disk error data
If we look at the underlying raw data with netstat -k (an undocumented option) we can see that iostat is reporting the data directly from the kstat with just a little formatting.
