Disk error detection

Unix Insider |  Storage Add a new comment

Q: I sometimes see messages on the console when a disk has an error. How can I automatically look for a disk that is giving soft errors before it fails completely? Are there any tools that will do this for me?

A: Traditionally, the disk device driver reacts to a problem by printing an error via syslog that gets printed on the console and stored in the /var/adm/messages file. While this is useful, it's an inefficient way to report a problem, and it's difficult to ensure that messages aren't lost or ignored. It's also hard to parse the error messages, because you might not know all the possible error conditions that could be reported.

The SyMON 1.x and 2.x products include a log file monitor that watches the /var/adm/messages file and warns the user if it sees certain kinds of errors. A new way to monitor disks was added in Solaris 2.6, and the iostat command was extended with -e and -E options that report error counts. I've now also extended the SE toolkit to look at the same information as part of the disk monitoring rule.

This isn't normally considered a performance question, but disks that are doing multiple retries due to transient errors can cause hard-to-find performance problems -- and the performance of a dead disk is zero!

Monitoring errors with iostat

Since Solaris 2.6, a new kstat data structure has been maintained in the kernel for every disk. Here is an example output from an Ultra 2 with two disks and a CD-ROM. The disk name is given in the normal device form of sd1 unless the -n option is also used to translate the name to c0t1d0s0 form.

The first line gives the total number of soft errors, hard errors, and transport errors; then the device identification string is broken down into vendor, product, firmware revision, and serial number. This string does not always work out properly, as you can see for the CD-ROM, where the serial number appears to be a date. The format used is vendor specific and difficult to parse in a general way. Next the size is given, and a detailed error count breakdown showing the total number of errors since boot is provided. In this case the system has been powered down using the cpr(7) capability. Each time this occurs, the disks are left spun down until they are accessed, giving rise to a No Device error for each power up resume.

  
% iostat -E

sd0     Soft Errors: 0 Hard Errors: 7 Transport Errors: 0 
Vendor: SEAGATE  Product: ST32550W SUN2.1G Revision: 0418 Serial No: 04406244 
Size: 2.13GB <2127708160 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 7 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 

sd1     Soft Errors: 0 Hard Errors: 7 Transport Errors: 0 
Vendor: SEAGATE  Product: ST32550W SUN2.1G Revision: 0418 Serial No: 04620031 
Size: 2.13GB <2127708160 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 7 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 

sd6     Soft Errors: 0 Hard Errors: 1 Transport Errors: 0 
Vendor: TOSHIBA  Product: XM-5401TASUN4XCD Revision: 1036 Serial No: 04/12/95 
Size: 18446744073.71GB <-1 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 1 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 

Underlying disk error data

If we look at the underlying raw data with netstat -k (an undocumented option) we can see that iostat is reporting the data directly from the kstat with just a little formatting.

    Add a comment

    Post a comment using one of these accounts
    Or join now
    At least 6 characters

    Note: Comment will appear soon after you have activated your account.
    Obscene/spam comments will be removed and accounts suspended.
    The information you submit is subject to our Privacy Policy and Terms of Service.

    ITworld LIVE

    StorageWhite Papers & Webcasts

    White Paper

    AppAssure vs Acronis

    In this study of data protection for environments with virtual and physical servers running Windows, openBench Labs tested AppAssure Backup and Replication software v 4.7 and Acronis Backup & Recovery 11. Both solutions utilize block-based technology to unify data protection operations.

    White Paper

    Guaranteeing 100% Backup Recovery

    The single biggest challenge for IT personnel involved in the data protection process is making sure that their backups are recoverable every time. Management and users won't remember the ninety-nine successful recoveries but they will always remember the one failure.

    White Paper

    ESG Analyst White Paper - VMware's vSphere Storage Appliance: High Availability for Small IT Operations

    Learn how small and midsized businesses are increasingly adopting virtualisation to deliver consolidation, improve data back up and disaster recovery and increase security with an in-depth new paper from the Enterprise Strategy Group (ESG). Learn directly from your peer's experiences and see why VMware's solutions are perfect for the growing and ambitious business.

    Webcast On Demand

    Understand Your Data: The Future of Backup and Archiving

    Archiving and Backup are the foundation of the next generation of information governance. However, commodity data protection tools and basic archives are only good for storing data. In the changing IT landscape, understanding what you are keeping, when to delete, and delivering insight to the business from your data is the future of these systems. Join us to hear the impact of private and public cloud solutions, "big data" and your choices while market evolves.

    Sponsor: Autonomy

    White Paper

    NetVault: #1 in the 2011 Oracle Backup Solutions Buyer's Guide

    Want to know how NetVault Backup compared against other Oracle backup software solutions - and why it's DCIG's #1 choice? In this 37-page report you'll get unbiased, third-party evaluations of Oracle backup software - and why NetVault Backup sits on the top of the list. Download your copy today.

    See more White Papers | Webcasts

    Ask a question

    Ask a Question