Q: How can I tell which processes are causing problems and which ones are stuck in a bottleneck?

A: A significant amount of data is available that is not shown by the ps command. In
addition, there are more clever ways to process and display data than top or proctool use. A new extension to the SE toolkit implements some of my ideas in this area. Along the way it becomes clear that the CPU usage measurements everyone relies on are somewhat inaccurate.

Process data sources

I described process data sources in my August 1996 Performance Q&A
column, but this time I'll go a step further with the data.
These data structures are described in full in the proc(4)
manual page. They are also available in the SE toolkit, so if you want
to obtain the data and play around with it, you should look at the code
for ps-ax.se and msacct.se.

The interface to /proc involves sending ioctl
commands or opening special pseudo-files and reading them (a new
feature of Solaris 2.6). The data that ps uses is called
PIOCPSINFO. Here's what you get back from
ioctl (you get slightly different data if you read it from
the pseudo-file):

proc(4)                   File Formats                    proc(4)
  PIOCPSINFO
     This returns miscellaneous process information such as that
     reported by ps(1). p is a pointer to a prpsinfo structure
     containing at least the following fields:
     typedef struct prpsinfo {
       char        pr_state;    /* numeric process state (see pr_sname) */
       char        pr_sname;    /* printable character representing pr_state */
       char        pr_zomb;     /* !=0: process terminated but not waited for */
       char        pr_nice;     /* nice for cpu usage */
       u_long      pr_flag;     /* process flags */
       int         pr_wstat;    /* if zombie, the wait() status */
       uid_t       pr_uid;      /* real user id */
       uid_t       pr_euid;     /* effective user id */
       gid_t