EZ monthly sar reports

Most admins monitoring system performace with sar (sysstat) collect performance data over the span of a week or two, but you can easily extend this to four weeks. In fact, you can extend the coverage period to well beyond a month if you need to maintain a longer view of performance and if you have enough disk space to hold all the data. Today's post details how to configure the coverage period and provides a script that will provide daily averages for as many days' of /var/log/sa## files you retain.

To change the number of days that sar will maintain data files, go to the /etc/sysconfig directory and edit the sysstat file. Change HISTORY=7 to HISTORY=28. If you change that number to anything greater than 28, you will end up with your sa## files separated into folders by month.

# How long to keep log files (days), maximum is a month
HISTORY=7

If you keep more than 28 days of performance data, sar will create a directory for each month. If just started collecting data with HISTORY set to greater than 28, you would see something like this.

drwxr-xr-x. 2 root root 4096 Aug 19 19:11 201208
lrwxrwxrwx. 1 root root   11 Aug 19 19:11 sa19 -> 201208/sa19

Notice that a directory has been set up for August 2012 in year/month format. A symbolic link associates the current collection (August 19th) to the data file in the 201208 directory. So /var/log/sa/sa19 is still a valid reference.

If you intend to hold performance data on line for a long period of time, take the size of one day's or one week's collection and multiply by the number of days or weeks to get the overall size. The data files should all be the same size unless your weekend collections are more sparse.

Today's script below is similar to one that I presented a couple weeks ago, but it will run against any number of /var/log/sa/sa## files. Plus, it doesn't care about the day of the week. Instead, it gets the date for each data file from the files themselves and displays these dates in the 08/19/2012 format.

#!/bin/bash
#
# summarize daily performance data

cd /var/log/sa

# ------------
# memory usage
# ------------
echo
echo "Memory & Swap"
echo "============="
echo -n "           "
sar -r -f $file | head -3 | tail -1 | awk '{print substr($0,12,100)}'
for file in `ls -tr sa* | grep -v sar`
do
    dt=`sar -r -f $file | head -1 | awk '{print $NF}'`
    echo -n $dt
    sar -r -f $file | tail -1 | sed "s/Average:  //"
done

# ------------
# load
# ------------
echo
echo "System Load"
echo "============="
echo -n "           "
sar -q -f $file | head -3 | tail -1 | awk '{print substr($0,12,100)}'
for file in `ls -rt sa* | grep -v sar`
do
    dt=`sar -q -f $file | head -1 | awk '{print $NF}'`
    echo -n $dt
    sar -q -f $file | tail -1 | sed "s/Average:  //"
done

# ------------
# CPU Usage
# ------------
echo
echo "CPU Usage"
echo "============="
echo -n "           "
sar -u -f $file | head -3 | tail -1 | awk '{print substr($0,12,100)}'
for file in `ls -tr sa* | grep -v sar`
do
    dt=`sar -u -f $file | head -1 | awk '{print $NF}'`
    echo -n $dt
    sar -u -f $file | tail -1 | sed "s/Average:  //"
done

Here is an example of what the data might look like if we'd been collecting since the beginning of the current month and had data files up to the 8th. In this example, we are only reporting on three types of performance measurements from the data available:

Memory & Swap
=============
            kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
08/01/2012     486528  32453580     98.52    291310  16291925   1064044   1033100     49.26    936512
08/02/2012     486497  32453611     98.52    309682  16275590   1064044   1033100     49.26    936512
08/03/2012     487076  32453032     98.52    305501  16282480   1064044   1033100     49.26    936512
08/04/2012     491773  32448335     98.51    312822  16269869   1064044   1033100     49.26    936512
08/05/2012     367462  32572646     98.88    334573  16469293   1064044   1033100     49.26    936512
08/06/2012     422750  32517358     98.72    737239  15490014   1064044   1033100     49.26    936512
08/07/2012     469591  32470517     98.57    352373  16080744   1064044   1033100     49.26    936512
08/08/2012     491786  32448322     98.51    300389  16179809   1064044   1033100     49.26    936512
System Load
=============
              runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
08/01/2012          0       365      0.06      0.06      0.05
08/02/2012          0       366      0.07      0.06      0.05
08/03/2012          0       366      0.07      0.07      0.06
08/04/2012          0       366      0.07      0.07      0.06
08/05/2012          1       369      0.47      0.46      0.45
08/06/2012          0       365      0.02      0.02      0.01
08/07/2012          0       366      0.06      0.05      0.04
08/08/2012          0       366      0.06      0.06      0.05
CPU Usage
=============
                  CPU     %user     %nice   %system   %iowait    %steal     %idle
08/01/2012        all      0.52      0.00      0.06      0.04      0.00     99.38
08/02/2012        all      0.55      0.00      0.06      0.04      0.00     99.35
08/03/2012        all      0.57      0.00      0.06      0.05      0.00     99.32
08/04/2012        all      0.59      0.00      0.06      0.05      0.00     99.30
08/05/2012        all      2.61      0.00      0.17      0.77      0.00     96.45
08/06/2012        all      0.16      0.00      0.04      0.02      0.00     99.79
08/07/2012        all      0.51      0.00      0.06      0.02      0.00     99.42
08/08/2012        all      0.53      0.00      0.06      0.04      0.00     99.38

The script itself is fairly straightforward. While the code is repetitious, I've broken it into sections in order to provide headings for each section. In addition, the sar commands within each section call out the different performance measurements -- -r for memory stats, -q for load stats, and so on. You could duplicate one of these sections and modify the headings and sar options to add other measurements to your report.

As is, the script will only report on the files in /var/log/sa. If you are collecting more than 28 days' worth of performance data, you would have to modify this script to look into each directory. That shouldn't be too hard.

Insider: How the basic tech behind the Internet works
Join the discussion
Be the first to comment on this article. Our Commenting Policies