Providing hosting services and running your own servers means that you're constantly on guard for things going wrong and failing. Setting up robust server monitoring using a package like Nagios or Zabbix is extremely complicated and time consuming, in fact it's some people's entire job at larger companies. At a smaller company, you may still have these critical sites, applications, and servers, but no one on call or available in the middle of the night. That means that after you go to bed, if a server goes down, you won't know about it until morning.
That is the situation I'm in. We've got server monitoring tools configured and running, but when an outage is detected I get an email or SMS to notify me. Those do me no good since I leave my phone silent in my office at night and if I were to keep it in my bedroom with audible alerts for every email and SMS I get my wife would kill me. The feeling of uncertainty that this causes has been a problem for me for years now and there is no worse way to start your day than by waking up to a pile of alert emails, angry customers, and down servers.
It would be preferable to have some type of indication in the night that there was a problem. Then I could get up, however annoying that may be, and solve it before anyone ever notices. My idea was to use a Raspberry Pi B+ I've had sitting around to ping websites and web apps over WiFi and then alert me in some visual way that might wake me up (I'm a light sleeper). With no fans and just a plain old micro-usb port for a power supply, the Pi has no moving parts and is totally silent - perfect for a bedroom. Just add a USB WiFi dongle and some code to monitor the servers and you're good to go.
My first task was to program the Pi to monitor the sites. Although this is not full blown server monitoring, by choosing WAN accessible pages from each server to monitor, along with a response assertion, you can get a good amount of confidence and status reporting. For example, even if your database server isn't WAN accessible, checking a web page that makes a connection to that database for a success phrase can tell you that the DB server is running fine.
My Pi is running Raspbian, a trimmed version of Debian Linux optimized for the Raspberry Pi hardware. Knowing I would be programming against the Pi hardware, I began looking around for Python code on GitHub to give me a head start on the monitoring. This would also be my first crack at coding in Python. I started with this open source Python script from Core Software Group and forked it into a new project. This gave me a nice head start on the project since the script already checked for sites plus an assertion phrase based on a timer and then fires off an email if a server is down.
I decided that I wanted the Pi board to light up in a bright way when a down server was detected so I began researching my options. While the Pi community encourages you to assemble the hardware yourself and solder a prototype together, I had limited time available and opted for a pre-assembled add-on board from Pibrella. For about $20 I got a fully assembled board that plugs straight onto the Pi headers that has 3 LEDs, a buzzer, and a push button. As an even bigger bonus, the Pibrella has a free python library that makes activating the features trivial.
Now that I had a plan, I began modifying the python program to use the Pibrella board and to perform some other tasks I thought I wanted. My Pi now does the following:
- On boot, run the monitor.py application as a daemon
- On run, determine the Pi's current IP address obtained by DHCP and email it to me
- At the configured interval, make a request to each server in the list
- If any server fails more times that the configured threshold, send an email and trigger the alert function
- On alert, pulse all of the LEDs on the Pibrella board and set the buzzer buzzing at a low frequency (50Hz) so that it's quiet but noticeable
- On pressing the button on the Pibrella, reset the alarm and sleep for an hour
The email that is sent in step 4 contains the URL of the site that failed so you can get right to work fixing the problem.
To make the Pi even more bedroom friendly, I covered the power LEDs that are always on by default on the motherboard and Pibrella board with electrical tape to block out the light. I also wrapped the WiFi dongle in electrical tape since it flashed on network activity.
This has been working great so far, although I've yet to have an alert trigger in the night, which brings me to my next issue to tackle, confirmation that it's working. The next addition I'll be making (which I'll update on Github) is to send an e-mail out periodically just to say that the app is still running just fine.
The peace of mind I've been given with this fun little embedded project has been well worth the small cost and effort. It makes me feel much better when I know things are up and running at a glance in the night. My wife hasn't left me yet either. Any other suggestions for a cool feature I could add?