Unix sleuthing: One problem leads to another

When tracking down an unusual problem on a Unix system, you may encounter many turns and dead ends on the path toward the solution as some problems may have little connection to what first appears to be troubling your system. We're going to follow an unusually winding path from first appearance of a problem to its eventual solution.

What initially brought this particular problem to the attention of the sysadmins was that the collection of performance statistics (via sar) had stopped accumulating. The previous day's sar report had been fetched from all the other servers and stored on the server used to analyze activity across the network. This one server had dropped from sight.

"Why would sar have suddenly stopped collecting stats?", we wondered. One day it's working, the next it's not. We saw no evidence of anyone logging in on the day data collection had stopped nor any previous days going back a long way. In addition, nothing about the sar setup seemed to have changed.

We noticed right away that something was amiss with the sys user account -- the one used to collect performance stats. If we attempted to switch users to sys, we saw these errors:

fermion# su sys
su: Invalid GID
fermion# su - sys
su: Invalid project ID

These were errors I had never encountered before. In fact, I found it hard to imagine what would be required for a GID to be considered "invalid" and wasn't sure what "project ID" referred to.

As it turned out, the su problem was a separate issue. For some reason, the su executable was owned by an ordinary user instead of by root.

fermion:/var/adm/sa # ls -l /usr/bin/su
-r-sr-xr-x   1 jaydoe  staff      17328 May  2  2001 /usr/bin/su

The invalid GID and project ID errors, therefore, did not refer to the user that I was trying to switch to, but to the ownership of the su executable.

When we tried running the /etc/rc2.dS21perf script, we ran into a second problem. It appeared that there was a disk space issue.

fermion:/ # /etc/rc2.d/S21perf start
sadc: creat failed: No space left on device

Examining disk space on the system, however, this didn't appear to be the case.

fermion:/ # df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0     140991   53111   73781    42%    /
/dev/dsk/c1t0d0s6    1985367  701002 1224804    37%    /usr
/dev/dsk/c1t0d0p0:boot
                       12127    1743   10384    15%    /boot
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
/dev/dsk/c1t0d0s3    1985367 1184764  741042    62%    /var
swap                 2915816       8 2915808     1%    /var/run
swap                 2916188     380 2915808     1%    /tmp
/dev/dsk/c1t0d0s5    1985367    1223 1924583     1%    /opt
/dev/dsk/c1t0d0s7    61387645 34852018 25921751    58%    /export/home
/dev/dsk/c1t0d0s1     387183  202055  146410    58%    /usr/openwin

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine