Solving the iostat disk mystery

Q: I keep seeing disks that are lightly used, but have extremely large
service times. Disks are supposed to have an average seek time of about
10-20 milliseconds, so why do they often report over 100 milliseconds
when they can't possibly be overloaded? Why does this happen? Is it a
sign of a problem?

                                 extended disk statistics 
disk      r/s  w/s   Kr/s   Kw/s wait actv  svc_t  %w  %b 
sd2       1.3  0.3   11.7    3.3  0.1  0.1  146.6   0   3 
sd3       0.0  0.1    0.1    0.7  0.0  0.0  131.0   0   0 

--Worried in Weyauwega

A: This is one of those
recurring questions that everyone seems to ask
at one time or another. The short answer is that it can safely be
ignored, as the disks are so lightly used that they don't make a
difference to the performance of the system.

It's a rather unsatisfying answer, however, as it doesn't explain why the
large service times occur in the first place. There are several
theories circulating:

Is it a bug?

This possibility can be dismissed immediately, since it's been seen for many
years, and has even been reported as a bug and investigated. The
calculations in iostat are well tested and correct.

Is it caused by rounding error at low-activity levels?

This is what I thought the problem was for many years. It was only
when we used I/O tracing to look at near-idle disks that we found out
what was really going on. Rounding errors cannot explain the high
service times we see.

Is it caused by Energy Star and thermal recalibration?

Modern disks have minds of their own. If you stop using them for a
while they power off their circuitry, and can even be programmed to
spin down completely. Even when they are in use, they go through a
recalibration sequence every now and again. This keeps the heads
perfectly aligned even when temperature changes cause thermal
expansion. While it's true that these activities will increase service
times, they should be relatively infrequent. We might be able to find
this kind of access in an I/O trace. It should appear as an isolated
short distance seek that takes a long time.

Is it something to do with the filesystem?

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine