Dark data and the digital sense organ
You have heard of dark matter right? The weird stuff that does not reflect light that makes up, oh, X percent of the universe (apparently). Now imagine that you are a super-intelligent little green man (or woman) inhabiting some parallel digital universe. You are so advanced that you have the ability to see digital data the same way that us humans see light. You point your digital sense organ in the direction of the Earth...
What do you see? You look puzzled. Why so? Something does not add up you say?. Your computations regarding data volumes are showing strange results are they? There is a lot more data in transit than actually pops out at valuable human-facing end points?
Hmmm. What is this "dark data" that makes up X percent of the digital earth but that does not reflect any value? Data that you can see with your digital sense organs but we humans cannot?
I know of one major source. Filtered e-mail. Here is a pattern I am seeing over and over again. A human subscribes to some service that sends out e-mails. The human then (being human) forgets how to un-subscribe from said service. The e-mails keep coming and clogging up and already clogged mailbox. What to do? Filter the e-mail. Bingo.
The annoying e-mail no longer hits the inbox. It is simple downloaded and immediately deleted...Congratulations, you have just created a dark data transfer. It is there, it consumes bandwidth and other resources yet no human ever sees it.
Sound familiar? I have a question. What percentage of the world e-mail traffic is dark? Is it a growing ercentage? I suspect so. How many small towns in the mid-west could be powered from the electricity wasted pointlessly shoving digital data down wires only to delete it immediately at the far end?
I have a second question. What can we do about it? Is this an inevitable consequence of the way e-mail works? A topic I wrote about before here?
Build your tech library with our book giveaways.
Windows PowerShell 2.0 Unleashed
By Tyson Kopczynski, Pete Handley, Marco Shaw; Published by Sams
Windows PowerShell Unleashed will not only give you deep mastery over PowerShell but also a greater understanding of the features being introduced in PowerShell 2.0–and show you how to use it to solve your challenges in your production environment. Enter now!
Ubuntu Server Administration
By Michael Jang; Published by McGraw-Hill Osborne Media
Realize a dynamic, stable, and secure Ubuntu Server environment with expert guidance, tips, and techniques from a Linux professional. Ubuntu Server Administration covers every facet of system management -- from users and file systems to performance tuning and troubleshooting. Enter now!
