Unix Tip: Shuffling file systems with rsync

I recently found myself needing to move several very large file systems into roomier data quarters on a disk array and began thinking about the most efficient
and reliable way to get the job done. The file systems just happened to be critical file systems containing hundreds of gigabytes of valuable source code and data files and I needed to be sure that the new file systems would be identical to the originals in both content and other parameters (file ownership and permissions) before making the new copies the file systems operational and reusing the original disks for other purposes.

There are, of course, many ways to copy files -- even very large collections of files -- from one place to another on a single system or across a network. The syntax for tar-to-tar operations had once been so critical a part of my systems management routine that I can still recite the basic "tar cvpBf - * | (cd /newdir; tar xBf -)" syntax in my sleep. Even so, as useful as tar-to-tar operations have proven themselves in the past, the syntax now seems overly clumsy and there are some disadvantages. For one thing, there's no option for encryption between systems. For another, the awkward syntax invites typing mistakes. In addition, tar-to-tar commands don't provide any way of verifying that the files have completely and accurately made it to their destination.

A number of other file copy options, therefore, have considerably more appeal.

While recursive scp commands perform well and reliably, they have one side effect that could cause a lot of trouble, including an increase in the size of the file system copies: on some systems -- such as Solaris, scp fails to preserve symbolic links and, instead, copies the contents of the target files.

With these thoughts in mind, I decided to use a tool that could both copy my files and verify that the files made it to their new destination intact -- rsync.

For the copy part of the operation, I used rsync with a simple set of options. The command .rsync .avz <source> <dest> packs a lot of punch into a few keystrokes. The -avz arguments specify that the files will be transferred in "archive" mode. This ensures that symbolic links, attributes, permissions, ownerships and such are all preserved in the transfer. For file systems in which symbolic links are used heavily, the failure to preserve the files as links could amount to a lot of storage overhead.

This rsync command results in a newly populated /newlocation/data directory. Like scp, this rsync command runs silently.

# rsync -avz /orig/data /newlocation

Any initial migration involving hundreds of gigabytes of data is going to take a considerable amount of time regardless of the tool used, but rsync seems to do nearly as well speed-wise as any tool I've tried.

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine