October 04, 2012, 10:48 AM — By now it’s kind of a truism that big data is going to be big business.
Marketing companies want to comb big data sets to determine what you’re likely to buy, so they can show you the right offers at the right time. Health care providers want to analyze big data so they can determine when and where outbreaks might happen and stop them before they become virulent. Various three-letter agencies based in DC want to mine that data so they can determine who the bad guys are before somebody decides to blow something up. The list is almost endless.
The problem with big data is, well, it’s big. It’s everywhere. It’s in multiple locations and incompatible file formats, with the same information duplicated in hundreds of different ways. Making sense of it can be a nightmare from which many organizations may never awake. But it doesn’t necessarily have to be that way.
I just had a fascinating discussion with a company called Chiliad (pronounced Kiliad, from the Greek word for a “group of one thousand”). Chiliad’s Discovery/Alert 7.0 software allows organizations with massive amounts of data spread across multiple locations to pretend that it’s all in the same place and in the same format, making it easier to search.
Without getting too heavily into the nitty gritty, Chiliad works by installing a
small appliance software node on every network where the data resides. Each appliance indexes all the data that’s available from each source, and communicates with the other appliances on the other networks. Chiliad’s software lets you use English language queries to get at this data and refine your searches. It then flags correlations between different bits of data and brings the relevant connections to the surface – leaving the data exactly where it is, without changing, converting, or even cleaning it.
So, let’s say you’re searching a medical database to find out if Vitamin D is helpful in treating metabolic syndrome. (Something I do at least once a week.) Chiliad’s results may tell you that it is, but they can also point out that Vitamin D is helpful in combatting diabetes and high blood pressure as well – something that may not have been otherwise obvious. And the software can do it virtually instantaneously (at least it did in the demo, from which this screen shot is taken).