August 12, 2008, 10:53 AM — You have heard of dark matter right? The weird stuff that does not reflect light that makes up, oh, X percent of the universe (apparently). Now imagine that you are a super-intelligent little green man (or woman) inhabiting some parallel digital universe. You are so advanced that you have the ability to see digital data the same way that us humans see light. You point your digital sense organ in the direction of the Earth...
What do you see? You look puzzled. Why so? Something does not add up you say?. Your computations regarding data volumes are showing strange results are they? There is a lot more data in transit than actually pops out at valuable human-facing end points?
Hmmm. What is this "dark data" that makes up X percent of the digital earth but that does not reflect any value? Data that you can see with your digital sense organs but we humans cannot?
I know of one major source. Filtered e-mail. Here is a pattern I am seeing over and over again. A human subscribes to some service that sends out e-mails. The human then (being human) forgets how to un-subscribe from said service. The e-mails keep coming and clogging up and already clogged mailbox. What to do? Filter the e-mail. Bingo.
The annoying e-mail no longer hits the inbox. It is simple downloaded and immediately deleted...Congratulations, you have just created a dark data transfer. It is there, it consumes bandwidth and other resources yet no human ever sees it.
Sound familiar? I have a question. What percentage of the world e-mail traffic is dark? Is it a growing ercentage? I suspect so. How many small towns in the mid-west could be powered from the electricity wasted pointlessly shoving digital data down wires only to delete it immediately at the far end?
I have a second question. What can we do about it? Is this an inevitable consequence of the way e-mail works? A topic I wrote about before here?