April 06, 2005, 8:51 AM — I recently had a chance to set up and then work with a console server. This was a first for me and so useful a device that I've become convinced that any systems administrator with more than a dozen systems to manage should know something about how console servers work and the service they can provide.
A console server is pretty much like what it sounds like. It's a device that consolidates console access to as many servers as the unit has ports. The particular unit that I set up and use is a Cyclades AlterPath with 48 serial ports, often referred to as an ACS48. With this box set up in the lab in which I work, 48 servers -- all "headless" (lacking consoles) -- are now accessible in single user mode or otherwise console-only states.
How are Servers Connected?
The console server, which has the general apearance of a router or switch, occupies one unit of rack space and connects to the local network in the same manner as all the other network nodes. Each of its 48 serial ports connects through a serial cable to the console (serial) port on a server. If a server is in a rack at the far end of the lab, the serial cable might be run through a set of patch panels. In other words, one serial cable might go from the console port on the server to a nearby patch panel and another might go from a patch panel near the console server to a particular port on the console server. Whether directly connected or not, the serial cable needs to be a "crossover" cable, similar to the null modem cables I used to build as needed.
How is Access Provided?
After associating every port on the new console server with a unique IP address, I could then open an ssh session to any of the assigned IP addresses and, after providing a username and password, I could work on the server console -- whether I was sitting in my office or working from another building, even from home.
The connection between me and the console server is always a TCP/IP connection while the connection between the console server and the target console is serial. This allows a large distance to exist between me and the console -- as long as nearly all of it is between me and the console server (serial connections are subject to length limitations that constrain the distance between a console server and the console ports that it supports).
How Do Console Servers Help?
Within a few days of setting up the new console server, I had two great opportunities to make use of it. In the first of these, the target server was sitting at the ok prompt, waiting for an fsck to clear up some file system damage. I logged in to the console and issued the required "fsck -y" and ^D and had the system up without having to leave my desk and my still-hot cup of decaf.
In the second situation, a user posted a ticket claiming that he couldn't access one of the servers and that he wasn't getting a response to his ping commands. While the obvious first guess was that the system had crashed, connecting to the system's console showed me that the system was up and running with no obvious signs that anything was wrong (ifconfig and routing information looked proper) except that the primary network interface was clearly not pushing or responding to packets. I had the system rebooted and working properly a couple of minutes later. Before the console server was in place, it would have taken me ten times that long to grab a laptop and a serial cable, find the system in its rack, reach through the snarl of cables inside the rack, fumble with the cable until I had clicked it into place in the serial port and then proceeded to deal with the problem.
Under the Hood
The console server is anything but a "dumb" device. In fact, it runs a version of embedded Linux that goes by the name "BusyBox" and runs a special-purpose web service. This means that I can configure the console server by logging in to the box's web server and that I can also log in to the box using ssh.
Console server configuration is dome exclusively through the auspices of the web site. I can give non-admin accounts to development staff on the rare occassions in which they need console access to install software and I can monitor (and, if necessary, kill) ongoing connections.
When logged into the console server with ssh, I can view the running processes (such as the web server) and, using ifconfig, I can verify the IP addresses assigned to each of the device's serial ports. I can also reboot the server should I need to.
