As a species, we generate some unconscionable amount of new digital data every day. The number is so big that no member of our species, that I know of, really has a good feel for the number. To try to wrap our heads around it we resort to desperate analogies e.g. "if we were to print it out in 10 point type on US legal paper the stack of paper would reach Neptune". Or, if every kilobyte of data was a blade of grass on a football field we would need more football fields than there are grains of sand on the beaches of the Earth"...
I just made those two up, by the way. I have no idea how far off the mark they are. For my present purposes, it doesn't really matter. Lets just agree the undeniable fact that the number is BIG. Now, how much of it is useful? If you needed to triage all the worlds daily data generation and decide what to keep and what to throw away, how would you proceed?
I have been thinking about this for a while now and have arrived at a disturbing conclusion. Namely, if you cast the net wide enough in terms of data consumers, ALL of the worlds daily data generation is equally important. Throwing any of it away is a hostage to fortune. For example, lets say you and I are collaborating on a project and sending documents to and fro. If we are working in the private sector, some of it is most likely not ours. There is a corporate entity that owns it and thus we need to conform to various corporate data retention rules. If we are working in the public sector, a similar but different set of strictures apply. What if we are just two citizens of the world shooting the breeze? Well, maybe we are working on some world changing concept and our "letters of correspondence" will be of historic interest in 100 years? Maybe our correspondence is subject to some scrutiny? Perhaps a random spot-check by a lawful security agency? Maybe we are but parts of larger teams and we want to ensure that team members working with us have full access to what we are up to?...
The list goes on. Maybe our choice of words will be of interest to linguists boldly looking to re-instate the ungrammaticality of the split infinitive? Maybe the timing of our exchanges will provide raw material for ground-breaking research into social interaction patterns...An infinity of maybe's.
What to do? Everything looks as if it might be useful. It might not, but can you take the chance of not preserving it? By and large, I think the world has decided that it cannot decide and thus, all over the world we are storing everything that goes through various pipes and various storage devices.
I can understand why this is so but cannot help thinking that we are engaged in perhaps the biggest punt in the history of humanity. We really have no idea how much of today's data will be intelligible tomorrow, even if we have perfect copies of it. Furthermore, we do not have a great track record as a species in getting software to be intelligent enough to trawl data looking for the interesting bits. We are much better at storing stuff than we are at retrieving stuff...
So we keep shovelling. Throwing more and more data onto storage devices that we will try to read in the future. Hoping against hope that by the time we get there, we will know what to look for and how to get it.
Maybe we are wasting our time and money? The failed science fiction writer in me sees another way of punting this. If all the digital data in the world is ultimately electromagnetic broadcasts then it is being recorded in the fabric of the universe anyhow. Maybe we should forget about our silly tapes and revolving disks and just rely on the ability of future humans to reach out into the space/time multi-verse and retrieve it all? Using the universe itself as the backup device.
Extremely far fetched for sure but I cannot help thinking that some of our current notions of storage/retrievability of magnetic media and file formats, say, 100 years from now are pretty far fetched also. That said, there is nothing else for us to do. We know what we don't know and given that, I see no alternative but to keep on, keeping on. Knowing that some of our efforts will be in vain but having no option but make the effort nonetheless.