Scrounging for Disk Space

Server disks are getting larger all of the time, but
this simple fact doesn't seem to be making much of a
difference in the system administrator's day-to-day
routine. OUr disks are still filling up faster than
most of us can trim them down. So, when we just need
to free up a chunk of disk space in a hurry on some
particular file system, what should we do?

Finding a chunk of disk space that we can potentially
move from one partition to another in order to create
needed space sounds easy enough, but is a relatively
tedious task. There are no obvious tools or commands
to identify candidate files or directories quickly and
easily. Aside from tracking down core files and large
temporary files (files that we can generally identify
by name), there's no easy way to find a file or group
of files that can be moved or removed in a pinch.

Let's look at some tricks that might make the task a
little easier.

Finding Large Files

To find large files within a particular partition, we
can always use a command like this:

find / -mount -type f -size +10000 -ls

This command will look through the root partition (but
NOT other partitions) for files larger than 5 MB or so,
the 10000 in this command representing the file size in
512-byte blocks and the "+" indicating "larger than".
The "mount" argument in this command keeps this find
command from searching through all other mounted
partitions for candidates while the "-type f" clause
keeps the command from reporting on anything other than
regular files.

Similar commands can be constructed to find large files
in other file systems. Here's one looking in /opt for
files larger than half a gigabyte that has found two 1
GB files:

# find /opt -mount -type f -size +1000000 -ls
  177 1049096 -rw------t  1 root     other    1073741824
Oct  7 16:05 /opt/swapfile
  178 1049096 -rw------t  1 root     other    1073741824
Oct  7 16:08 /opt/swapfile2

NOTE: In this example, the located files appear to be
swap files. We can determine if they are in current use
as swap with a "swap -l" command.

Finding Large Directories

If we don't find an adequate number of large files that
we can move or remove from the target file system, we
might be able to find a directory that, moved intact, will
create the needed space. Candidate directories are
trickier, though, because we need to determine their size
based on their contents, not on the size of the directory
files themselves. Sure, we can use a combination of find
and "du -sk", but exactly how to put these commands
together to create useful output presents something of
a challenge -- and one that I've found myself having to
address several times a year.

Instead of coming up with an ad hoc command for finding
candidate directories on as as needed basis, let's
examine a Perl script that solves the problem once and
for all. This script uses a find command that is similar
to those shown above, but adds a "du -sk" command to
compute the size of each directory and then a selection
process based on this size to determine which of the
directories are candidates and should be reported.

--------------------------- cut here ---------------------------

#!/bin/perl -w
#
# find big directories in a particular file system

# ---- prompt user ----
print "directory> ";
$partName=;
chomp $partName;
# ---- prompt user ----
print "size (in KB)> ";
$dirSize=;
chomp $dirSize;

# ---- find directories ----
@dirs=`find $partName -mount -type d | xargs du -sk`;

# ---- print details on large directories ----
for $_ ( @dirs ) {
    chomp;
    my($size,$dirname) = split;
    if ( $size > $dirSize ) {
        if ( $size > 1000000 ) {        # convert to GB
            $GB=$size/1000000;
            printf "%8.2f %s %s", $GB, "GB", "$dirname\n";
        } elsif ( $size > 1000 ) {      # convert to MB
            $MB=$size/1000;
            printf "%8.2f %s %s", $MB, "MB", "$dirname\n";
        } else {
            printf "%8.2f %s %s", $size, "KB", "$dirname\n";
        }
    }
}

--------------------------- cut here ---------------------------

This script prompts the user for the directory to be searched.
The response can be the mount point for a file system or any
subdirectory. In any case, the search will be constrained to
the particular partition. It also prompts the user to enter a
number that corresponds to the amount of disk space that the
user wants to locate. If the user enters 1000, for example,
this indicates that he/she wants to find directories that are
larger than 1 megabyte.

The script issues a find command with a pipe to an xargs command
that runs a "du -sk" command and stashes the results in an array.
It then runs through the array looking for directories which
exceed the size specified and prints formatted output. Perl's
printf command is used to limit the number of decimal places
displayed and directory sizes are converted to gigabytes or
megabytes to keep the user from having to count digits in order
to grasp the reported sizes. Here is some sample output:

# ./findBigDirs
directory> /export/home/shs
size (in KB)> 100000
 1.93 GB /export/home/shs
983.15 MB /export/home/shs/sw
185.66 MB /export/home/shs/sw/jrun4
185.66 MB /export/home/shs/sw/jrun4/jrun4_install
155.77 MB /export/home/shs/sw/jrun4/jrun4_install/HotFixes
343.29 MB /export/home/shs/sw/Customized
224.33 MB /export/home/shs/sw/Customized/JRun4
126.86 MB /export/home/shs/sw/Customized/JRun4/Packages
380.47 MB /export/home/shs/sw/Sun_Patches
290.44 MB /export/home/shs/sw/Sun_Patches/8_Recommended
303.60 MB /export/home/shs/Java

In this example, the script was asked to look for directories
larger than 100 megabytes. It found eleven of them though.
obviously, some are contained within others in the list.
The relatively short list of candidate directories is useful.
I can quickly see that, were I to move my sw/Sun_Patches
directory, I would recover just over 380 megabytes of disk
space in the /export/home partition. I can also see that my
sw (software) directory is using nearly half of the space
occupied by my home directory.

Where is Disk Space Going?

A simple Perl script such as the one shown can also be used to
give us an idea where disk space in a growing file system is
going. If we simply want to answer a question such as
"How are we using up SO much disk space?", we could look
through all of /export/home (or all of some other file system)
for directories larger than a gigabyte and then, within the
reported directories, look for directories larger than 100
megabytes. Getting a quick idea of which directories are
taking up a lot of space, we can make informed decisions about
our strategy for cleaning up and managing our files.