Unix Tip: Shuffling file systems with rsync

I recently found myself needing to move several very large file systems into roomier data quarters on a disk array and began thinking about the most efficient
and reliable way to get the job done. The file systems just happened to be critical file systems containing hundreds of gigabytes of valuable source code and data files and I needed to be sure that the new file systems would be identical to the originals in both content and other parameters (file ownership and permissions) before making the new copies the file systems operational and reusing the original disks for other purposes.

There are, of course, many ways to copy files -- even very large collections of files -- from one place to another on a single system or across a network. The syntax for tar-to-tar operations had once been so critical a part of my systems management routine that I can still recite the basic "tar cvpBf - * | (cd /newdir; tar xBf -)" syntax in my sleep. Even so, as useful as tar-to-tar operations have proven themselves in the past, the syntax now seems overly clumsy and there are some disadvantages. For one thing, there's no option for encryption between systems. For another, the awkward syntax invites typing mistakes. In addition, tar-to-tar commands don't provide any way of verifying that the files have completely and accurately made it to their destination.

A number of other file copy options, therefore, have considerably more appeal.

While recursive scp commands perform well and reliably, they have one side effect that could cause a lot of trouble, including an increase in the size of the file system copies: on some systems -- such as Solaris, scp fails to preserve symbolic links and, instead, copies the contents of the target files.

With these thoughts in mind, I decided to use a tool that could both copy my files and verify that the files made it to their new destination intact -- rsync.

For the copy part of the operation, I used rsync with a simple set of options. The command .rsync .avz <source> <dest> packs a lot of punch into a few keystrokes. The -avz arguments specify that the files will be transferred in "archive" mode. This ensures that symbolic links, attributes, permissions, ownerships and such are all preserved in the transfer. For file systems in which symbolic links are used heavily, the failure to preserve the files as links could amount to a lot of storage overhead.

This rsync command results in a newly populated /newlocation/data directory. Like scp, this rsync command runs silently.

# rsync -avz /orig/data /newlocation

Any initial migration involving hundreds of gigabytes of data is going to take a considerable amount of time regardless of the tool used, but rsync seems to do nearly as well speed-wise as any tool I've tried.

After migrating each of the file systems to their new locations, I then used rsync again to evaluate the success of the move. This may seem like overkill after using rsync to make the copy, but I wanted a nice tidy report showing that the files were in synch.

For verifying the contents of a new directory, rsync is the clear winner. While it's no speed demon for copying large file collections in the first place (especially if you use the verbose option), it beats the socks off any other tool I know of for efficiently comparing two sets of files and reporting any differences that might exist. Even the difference of a single byte in a single file is going to show up and, in normal conditions, be rectified.

Primarily used to synchronize files between two systems, rsync is both very efficient at doing this and extremely reliable. By computing and comparing files using highly reliable "rolling" checksums, rsync is able to accurately compare files while transmitting very little data. This makes the software an extremely good tool for efficient replication, but this same feature can also be applied to simply verifying that two collections of files are identical.

For the verification process, I chose to run rsync in dry-run mode in which it reports any file or byte transfers that it would make, but doesn't make any changes. If I had reason to expect files to have changed between the copy and the verification processes, I might not have included this option. On the other hand, I probably would have waited until the file systems were clearly quiescent before attempting to move them.

vHere's a sample rsync command for verifying a successful copy:

# rsync -avz --dry-run --verbose --progress --stats \
      --delete /orig/data /newlocation

The verbose, progress and stats arguments report what's going on while rsync is running. We'll see some of the output this produces shortly.

The --delete option in the second line will cause files which have been removed from the original file system to be removed in the copy as well. Along with the dry-run option, this command will tell you what it would delete, but not actually delete anything. Here's some sample output from a verification run:

building file list ...
2917 files to consider

Number of files: 2917
Number of files transferred: 0
Total file size: 84361224 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 24488
Total bytes sent: 24500
Total bytes received: 20

sent 24500 bytes  received 20 bytes  2335.24 bytes/sec
total size is 84361224  speedup is 3440.51

In this example, it's those "Number of files transferred: 0" and "Total transferred file size: 0 bytes" lines that tell you nothing would need to be done to synchronize the original and copy of the file collections. In other words, rsync found that all of the files on the source were also on the destination system and that the files on the two systems were identical.

Dry-run will compare the contents of the original and copied file systems and report anything it would change if it were running the operation "for real". In the report shown below, rsync found a single file which it would need to transfer for the two file systems to be synchronized.

building file list ...
7001 files to consider
/orig/activity.log

Number of files: 7001
Number of files transferred: 1
Total file size: 448198765 bytes
Total transferred file size: 438046 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 141728
Total bytes sent: 141744
Total bytes received: 24

sent 141744 bytes  received 24 bytes  31504.00 bytes/sec
total size is 448198765  speedup is 3161.49

The output above shows that one file (438,046 bytes) would be transferred if we were not in dry-run mode. This comparison operation took only about 4 seconds to run.

You could also run rsync in its normal proactive mode, updating the destination system automatically whenever rsync finds a discrepancy such as this. This is its normal operating mode. The choice depends on whether you expect differences and whether you want to intercede before forcing one system to look just like the other.

I have found rsync to be both easy to use for copying files and incredibly efficient in the verification process. When I think back to the days when I judged the success of a migration of a collection of files by determining that the size of the data collection copy was roughly equivalent to the size of the original, I realize how much reliability I've gained by using smarter tools. This marks an order of magnitude improvement in the process of replicating data
and one that saves me considerable worry whenever I have to move large amounts of critical data from one location to another.

Of course, any copying and verification of file systems are best performed when the file systems are quiescent.