Unix Tip: Using SCAT (Solaris CAT) for analyzing crash dumps
Want to do something with those crash dump files other than remove them? Want to extract some useful information without a lot of work? Take a look at Sun's free crash analysis tool, scat.
While "scat" in this context stands for "Solaris Crash Analysis Tool", I find myself drawing an analogy to the other meaning of the word. "Scat" refers to the dung of certain animals. Analysis of animal scat can tell you a lot about the health and the diet of the particular animals and can be useful in identifying the species as well. Similarly, scat analysis of the system variety can provide useful information for diagnosing the ailments of your Solaris servers -- in particular, what caused them to crash. Examining the "droppings" of a system can require considerable skill if you end up delving deeply into the crash dump files. At the same time, even the simplest use of the scat command can provide you with some of the most telling information about a crash. You simply give the tool the numeric identifier of the crash dump you want to know about. If your crash dump files are unix.0 and vmcore.0 (i.e., the first set of a possible collection of crash dump files), for example, you would issue commands such as these:
# cd /var/crash/boson # /opt/SUNWscat/bin/scat 0
Notice that, since the scat program wants only the numeric identifier for the crash dump files, you must first cd to the directory in which these files are stored.
The unix.X and vmcore.X files are generated when a Solaris system panics and are normally stored in /var/crash/`uname -n`. The initial output from scat will identify the system and provide information on the date and time the system crashed, how long the system had been up at the time, the system type, hostid, panic string, etc. Here's an example:
Solaris[TM] CAT 4.1 (build 526) for Solaris 9 64-bit SPARC(sun4u) Copyright © 2003 Sun Microsystems, Inc. All rights reserved. Patents Pending. Use is subject to license terms. Sun Microsystems proprietary - DO NOT RE-DISTRIBUTE! Feedback regarding the tool should be sent to SolarisCAT_Feedback@Sun.COM opening vmcore.0 ...dumphdr...symtab...core...done loading core data: modules...panic...memory...time...misc...done loading stabs...read_type_db: Wrong number of lines in database, or database doesn't end in a newline unable to load any stabs file patches... - NOT AVAILABLE (No such file or directory) done core file: /var/crash/boson/vmcore.0 user: Super-User (root:0) release: 5.9 (64-bit) version: Generic_112233-11 machine: sun4u node name: boson domain: lab.particles.org hw_provider: Sun_Microsystems system type: SUNW,Sun-Fire-V210 hostid: 837844c7 time of crash: Tue Apr 22 11:49:52 EDT 2008 age of system: 22 hours 5 minutes 4.48 seconds panic cpu: 0 (ncpus: 1) panic string: free: freeing free block, dev:0x200000016e, block:32032, ino:6057255, fs:/homes running sanity checks.../etc/system...ndd...sysent...misc...done SolarisCAT(vmcore.0)>
Build your tech library with our book giveaways.
Windows PowerShell 2.0 Unleashed
By Tyson Kopczynski, Pete Handley, Marco Shaw; Published by Sams
Windows PowerShell Unleashed will not only give you deep mastery over PowerShell but also a greater understanding of the features being introduced in PowerShell 2.0–and show you how to use it to solve your challenges in your production environment. Enter now!
Ubuntu Server Administration
By Michael Jang; Published by McGraw-Hill Osborne Media
Realize a dynamic, stable, and secure Ubuntu Server environment with expert guidance, tips, and techniques from a Linux professional. Ubuntu Server Administration covers every facet of system management -- from users and file systems to performance tuning and troubleshooting. Enter now!
GREAT article!! I always
GREAT article!! I always look forward to them, they are so imformative.You said: "However, it's not too hard to pick out additional evidence that the crash was due to a ufs failure. It takes considerable familiarity with the system calls included in the analyze output to identify the sequence as the removal of a large file"
But what was it that tipped you off to this?