The Left Field

Bootstrapping tomorrow, today

As a species, we generate some unconscionable amount of new digital data every day. The number is so big that no member of our species, that I know of, really has a good feel for the number. To try to wrap our heads around it we resort to desperate analogies e.g. "if we were to print it out in 10 point type on US legal paper the stack of paper would reach Neptune". Or, if every kilobyte of data was a blade of grass on a football field we would need more football fields than there are grains of sand on the beaches of the Earth"...

I just made those two up, by the way. I have no idea how far off the mark they are. For my present purposes, it doesn't really matter. Lets just agree the undeniable fact that the number is BIG. Now, how much of it is useful? If you needed to triage all the worlds daily data generation and decide what to keep and what to throw away, how would you proceed?

I have been thinking about this for a while now and have arrived at a disturbing conclusion. Namely, if you cast the net wide enough in terms of data consumers, ALL of the worlds daily data generation is equally important. Throwing any of it away is a hostage to fortune. For example, lets say you and I are collaborating on a project and sending documents to and fro. If we are working in the private sector, some of it is most likely not ours. There is a corporate entity that owns it and thus we need to conform to various corporate data retention rules. If we are working in the public sector, a similar but different set of strictures apply. What if we are just two citizens of the world shooting the breeze? Well, maybe we are working on some world changing concept and our "letters of correspondence" will be of historic interest in 100 years? Maybe our correspondence is subject to some scrutiny? Perhaps a random spot-check by a lawful security agency? Maybe we are but parts of larger teams and we want to ensure that team members working with us have full access to what we are up to?...

The list goes on. Maybe our choice of words will be of interest to linguists boldly looking to re-instate the ungrammaticality of the split infinitive? Maybe the timing of our exchanges will provide raw material for ground-breaking research into social interaction patterns...An infinity of maybe's.

What to do? Everything looks as if it might be useful. It might not, but can you take the chance of not preserving it? By and large, I think the world has decided that it cannot decide and thus, all over the world we are storing everything that goes through various pipes and various storage devices.

I can understand why this is so but cannot help thinking that we are engaged in perhaps the biggest punt in the history of humanity. We really have no idea how much of today's data will be intelligible tomorrow, even if we have perfect copies of it.

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world