If broadly deployed, tools like Chiliad’s could make this kind of data correlation as easy as using Google. In fact, if there were a consumer version of Discovery/Alert – or I had a few hundred thousand dollars to spare – I would want to use it on top of Google, so I could sort results to get the most up-to-date information on a topic (something Google is particularly bad at). I’d buy that in a heartbeat.
What’s interesting from a privacy perspective are Chiliad’s clients. For the past decade Chiliad has worked with the Department of Homeland Security and the FBI, helping the spooks sort through 15 billion records from more than 100 sources. The Feds use Chiliad to monitor all flights, trucks, cars, ships, and passengers entering this country. If you’ve ever passed through US Customs, your data has flowed through Chiliad’s software.
Beyond that, though, Chiliad couldn’t tell me what kinds of data the DHS is interested in. Even they don’t know all of the data sources the DHS combs through each day, and if they did, well… you know the rest of that joke.
Now Chiliad is moving into medical records, another area where the data is jumbled, massive, and highly sensitive, with lots of privacy landmines. The idea is that by allowing hospitals and pharmaceutical companies to search across patient records and research data from multiple sources, doctors may be able to pick out patterns that may not have been visible when looking at smaller data sets – like, say, seeing that a particular treatment is vastly more effective for a certain malady. And Chiliad’s auditing capabilities will allow admins to know exactly who saw what data and when they saw it, keeping it in compliance with regulations like HIPAA.
Chiliad marketing veep Ken Rosen says Chiliad’s experience working with the DHS makes it uniquely qualified to handle highly regulated personal data such as medical records.
The other problem with big data is that, if it’s misinterpreted, the results could be disastrous. Let’s say the DHS has analyzed a few petabytes of consumer purchase data and determined that terrorists like to order takeout pizza, pay cash for their groceries, take lots of cross-Atlantic flights, and visit Jihadist Web sites. If you happen to do all of these things – and you’re not a terrorist -- it’s conceivable you could be mistaken for one, based entirely on what the data seems to be saying about you.
Obviously, the NSA is not going to release the algorithm it uses to determine whether suspect A is a potential terrorist while suspect B is not. On the other hand, Rosen says the Feds take individual privacy quite seriously.