Respawning too fast

by Sandra Henry-Stocker

June 15, 2005 —

The screen was full of messages that read:

INIT: id "T0" respawning too fast: disabled for five minutes

Not used to seeing the server console covered with errors, no one on-site was sure how severe the problem was or what to do about it. In fact, no one knew whether respawning was a normal thing or what was being disabled time and time again. And no one knew where these messages were coming from. For the Unix sysadmins, the importance of the init process is clear as was the fact that this particular message was being generated by the init process. The "mother" of all processes, init is responsible for orchestrating the boot process and for determining what happens on a system whenever it is shut down or switched to another run state (such as singleuser).

Killing init is one of the time-proven big bad mistakes that can halt a system very unceremoniously. In the old days, this mistake usually involved typing "kill 1 1" as root when a sysadmin meant to type "kill -1 1". The latter command sends a hangup signal to the process, urging it to re-read its configuration file; the other slashes its virtual throat. Respawning is the opposite of killing. Respawning is the process of restarting a process when it fails for some reason. The T0 message, on the other hand, probably doesn't ring bells with many people. So let's look at what T0 refers to and where it's defined. To do this, let's first look a bit more deeply at init. The init process should be running all the time. It is the parent of many processes started at boot time. In the listing below, for example, the processes whose parent process (PPID) is 1 were started by init during the boot process and init itself appears in the third line of output.



> ps -ef
     UID   PID  PPID  C    STIME TTY      TIME CMD
    root     0     0  0   Apr 03 ?        0:00 sched
    root     1     0  0   Apr 03 ?        0:50 /etc/init -
    root     2     0  0   Apr 03 ?        0:00 pageout
    root     3     0  0   Apr 03 ?       143:16 fsflush
    root   350     1  0   Apr 03 ?        0:00 /usr/lib/saf/sac -t 300
    root   166     1  0   Apr 03 ?        0:00 /usr/lib/inet/in.ndpd
    root    60     1  0   Apr 03 ?        0:00 /usr/lib/sysevent/syseventd
    root   226     1  0   Apr 03 ?        0:02 /usr/lib/autofs/automountd

On many Unix systems, such as Solaris and Linux, init has a configuration file named inittab. This colon-delimited file details the script(s) that init will run when used to change runs states. If init is told to shut down a system with the command "init 0", for example, init will read the line that specifies what it should do when initiating run state 0. On a Solaris system, most of the lines in the inittab file will look like these two lines:

s2:23:wait:/sbin/rc2 >/dev/msglog 2<>/dev/msglog /dev/msglog 2<>/dev/msglog

The first field is a label. The second is the run states in which the particular line will be used. The fourth is the script that will be invoked. The third tells the init how to treat the process that it starts -- for example, whether to wait for it to finish or restart it if it stops (in other words, respawn it).

If init is starting run state 3, it will read both of these line (since 3 is included in field 2 in both of these lines) and run the script listed in the third field of each line (/sbin/rc2 and /sbin/rc3).

The "T0" in our error message is the label of yet another line in the inittab file. On the system reporting the T0 error, that line looked like this.

T0:2345:respawn:/sbin/mgetty ttyS0

In this line, we can see that when init starts run states 2, 3, 4 or 5, it will run the command /sbin/mgetty ttyS0. We also see that this particular inittab entry specifies "respawn" in the third field. This tells init that, if the process it starts dies, it should restart or "respawn" it.

The screenful of INIT warnings came about because the mgetty process wouldn't start. As a result, init was continually trying to respawn the process. Determining why the mgetty process might be running into problems is another topic altogether. Knowing why the screen is filling up with these messages relies on knowing how init works.

ITworld.com