Unix Tip: Reassembling very large "split" files

One of the unwritten rules of computing says that, as various storage media grow in size, so do the files that are written to them. I recently found myself staring at two files, each roughly 4 GB in size, that needed to be combined into one before the file could be decompressed and its contents extracted. What I was expecting to work with was a large cpio archive.

I had loaded each of the file components from a DVD, thus the reason that the file had to be compressed and then broken into two pieces, each of which could be written to a DVD.



The files had the extensions gzaa and gzab. This naming convention is meant to tell the recipient of the files that 1) the files are both chunks of a larger file that needs to be reassembled and 2) that the resultant file is gzipped. These names also suggest that the original file was broken into two pieces with the "split" command -- a convenient utility that breaks files into smaller pieces and names the pieces and appends "aa", "ab" and so on to give each chunk of the original file a distinct name.



To reassemble the file from its components, I used the cat command, basically adding the contents of the second file to the end of the first. While this may seem like an obvious solution, I had never done this before with compressed cpio files or with files of this size. The rebuild of the original file took a long time, but worked perfectly. I soon had a file nearly 8 GB in size named bigfile.gzaa.

# cat bigfile.gzab >> bigfile.gzaa

This concatenation operation took a long time to run. I was, after all, reading and writing nearly 4 GB of disk contents. Rejoining files with cat takes one less operation than files you have to join together. If I had two more files, I would have run these commands as well:

# cat bigfile.gzac >> bigfile.gzaa
# cat bigfile.gzad >> bigfile.gzaa

Some people prefer to first rename the original file (e.g., mv bigfile.gzaa to bigfile.gz) before starting the concatenations so that the file they are assembling has the intended name while it's being rebuilt. Others rename the file after the file has been rebuilt.



In my case, once bigfile.gzaa contained both chunks of the original file, I removed bigfile.gzab to recover the 4GB of disk space that it occupied and renamed bigfile.gzab to the more familiar form, bigfile.gz.



At this point, I could unzip bigfile.gz and recover bigfile -- the cpio file that I was expecting when I first mounted the DVDs. The unzipping of bigfile.gz also took a long time to complete (approximately half an hour I would guess). The cpio file was more than 13 GB. Without the gzip compression, it would have required four DVDs instead of the two that it occupied when I received it.



If you have a large file that won't fit on a single DVD, you can use the reverse of the process that I used to break your file into appropriately sized pieces and write your DVDs. The split command can create a series of 4 GB chunks with a command like this:

# split -b 4000m bigfile.gz bigfile.gz

The 4000m argument means 4,000 MB or 4 GB. The bigfile argument on the end of the line tells split to name your files bigfile.gzaa, bigfile.gzab and so on. Without this, split will give your files the simpler names aa, ab etc.



Working with very large files is not much different than working with smaller files as long as you have enough disk space, enough time and some insights into the files' format.

ITworld

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine