Unix sleuthing: One problem leads to another
When tracking down an unusual problem on a Unix system, you may encounter many turns and dead ends on the path toward the solution as some problems may have little connection to what first appears to be troubling your system. We're going to follow an unusually winding path from first appearance of a problem to its eventual solution.
What initially brought this particular problem to the attention of the sysadmins was that the collection of performance statistics (via sar) had stopped accumulating. The previous day's sar report had been fetched from all the other servers and stored on the server used to analyze activity across the network. This one server had dropped from sight.
"Why would sar have suddenly stopped collecting stats?", we wondered. One day it's working, the next it's not. We saw no evidence of anyone logging in on the day data collection had stopped nor any previous days going back a long way. In addition, nothing about the sar setup seemed to have changed.
We noticed right away that something was amiss with the sys user account -- the one used to collect performance stats. If we attempted to switch users to sys, we saw these errors:
fermion# su sys su: Invalid GID fermion# su - sys su: Invalid project ID
These were errors I had never encountered before. In fact, I found it hard to imagine what would be required for a GID to be considered "invalid" and wasn't sure what "project ID" referred to.
As it turned out, the su problem was a separate issue. For some reason, the su executable was owned by an ordinary user instead of by root.
fermion:/var/adm/sa # ls -l /usr/bin/su -r-sr-xr-x 1 jaydoe staff 17328 May 2 2001 /usr/bin/su
The invalid GID and project ID errors, therefore, did not refer to the user that I was trying to switch to, but to the ownership of the su executable.
When we tried running the /etc/rc2.dS21perf script, we ran into a second problem. It appeared that there was a disk space issue.
fermion:/ # /etc/rc2.d/S21perf start sadc: creat failed: No space left on device
Examining disk space on the system, however, this didn't appear to be the case.
fermion:/ # df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c1t0d0s0 140991 53111 73781 42% /
/dev/dsk/c1t0d0s6 1985367 701002 1224804 37% /usr
/dev/dsk/c1t0d0p0:boot
12127 1743 10384 15% /boot
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
/dev/dsk/c1t0d0s3 1985367 1184764 741042 62% /var
swap 2915816 8 2915808 1% /var/run
swap 2916188 380 2915808 1% /tmp
/dev/dsk/c1t0d0s5 1985367 1223 1924583 1% /opt
/dev/dsk/c1t0d0s7 61387645 34852018 25921751 58% /export/home
/dev/dsk/c1t0d0s1 387183 202055 146410 58% /usr/openwin
Symantec Backup Exec 12 and Backup Exec System Recovery 8 deliver industry leading Windows data protection and system recovery. Download this whitepaper to find out the top reasons to upgrade and how to get continuous data protection and complete system recovery.
Data and system loss — from a hard drive failure, malicious attack, natural disaster, or simple human error — can happen anytime. Don’t leave your business vulnerable. Make sure you have a secure recovery strategy in place. Symantec's latest backup and system recovery technology can efficiently restore critical applications, individual emails and documents and even restore your entire system in minutes in the event of a loss.
Businesses face a growing challenge to ensure that the IT environment is properly protected. Backup Exec 12 integrates with other applications in the Symantec family of products, to complement your current data protection strategy, keep your data securely backed up and make it recoverable when you need it most.
VMware ESX Server in the Enterprise
By Edward L. Haletky
Published Dec 29, 2007 by Prentice Hall.
Enter now! | Official rules | Sample chapter
Green IT
By Toby Velte, Anthony Velte, Robert C. Elsenpeter
To be published Oct. 10, 2008 by McGraw Hill Professional
Enter now! | Official rules | About the book
