strong>Summary: Many of us have had to deal with application crashes that are hard or impossible to reproduce, especially at software developers' sites. Yet it can be difficult to fix the problem without reproducing it. This feature describes a few tools to help you generate a traceback, a chain of all function calls that the application was executing at the time of the crash. A traceback will help locate the trouble or at least narrow it down. Later you can also collect crash statistics and use them to enhance the quality-control procedures.

When a Solaris application crashes, it usually produces a core file, which is a disk copy of the application's memory at the time of the crash.

One way to generate a traceback is to use a debugger such as dbx with the core file:

% dbx /path/executable core
(dbx) where > traceback.txt
(dbx) quit

In addition, starting with Solaris 8, the pstack(1) utility can also print out a traceback from the core file, just like dbx:

% pstack core > traceback.txt

If you can't use dbx or pstack with a core file for any reason, read on for a way of generating a traceback directly from the application.

Handling application core files

Generally, dealing with application core files is difficult for the following reasons:

• No debugger: Many user sites don't have access to a debugger such as dbx. It costs money, needs maintenance, and it is not really needed there in most cases.
• Big core files: Saving a core file generated by a large application can require a lot of disk space, which may not be available. Also, sending such core files to the software developers may be hard because of the core file's large size.
• Unusable core files: Interpreting the core files on a machine different from where the core file was created is often impossible because no two systems are exactly alike. There are always differences in hardware, OS versions, and patch levels, which can make the debugger refuse to read a core file from a different machine. Therefore, sending the core files to the software developer usually doesn't help.

Generating a traceback from the application

The solutions described below use pstack(1), which is specific to Solaris. However, the same methods can be used with other operating systems where similar functionality is available.

Note that generating a traceback on crash provides only one more piece of the puzzle in determining why the application has crashed and how to fix the problem. Nevertheless, such a traceback may lead to the underlying problem causing the crash, or at the very least to determining whether the problem is in the application or in the system. In any case, getting a traceback is a step in the right direction.

My recent article "Building Library Interposers for Fun and Profit" described how to build library interposers and use them for various debugging and performance tuning tasks. Some of the tools described in this article are additional applications of the library interposition technology.

If the application does not have signal

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine