March 25, 2013, 3:41 PM — For several years I've been talking to big data companies trying to sell products and to IT executives trying to get their hands around the issues. Some interesting problems persist. It's clear we're still at the beginning of understanding this problem, and we're likely still a long way from understanding the promise of using this information.
Companies such as Facebook and Google capture massive amounts of information. They generally get pounded for violating privacy, as neither they, nor we, can figure out what they are doing with this data. We assume they are using it against us, even though they very well may be trying to use it for our benefit.
There Are No More Dragons Protecting Your Data
Many of the issues that have historically revolved around large data repositories pertain to how you manage them. This chiefly means assuring that those who need access, for everything from management reports to compliance, can get the information they need when they need it. It also means assuring that data is stored safely. This historically has involved services from vendors such as Iron Mountain, where the data is often so safe that no one can figure out how to get it back out.
This speaks to the historic problem with managing data. We've treated it like pirate treasure, finding creative, inexpensive ways to bury it and coming up with equally creative excuses when can't get to it in a timely manner, if at all.
Oh, the booty exists, we're sure of that-but we don't know exactly where, and the really old data is often so poorly indexed and stored that it seems like we'd have been better off if we hadn't stored it in the first place.
Emerging public cloud resources promise inexpensive storage with the higher likelihood of future accessibility. Haphazard piles of treasure have been stashed in neat little rows, and a friendly elf has replaced the fire-breathing dragon. The only trade-offs, of course, are security, governance and compliance.