Big data skeptics, inside and out

'Question everything,' skeptics warn. That's already covered.

The clarion call for data scientists continues, but there is building pushback on whether the drive towards data-driven decision making is a good idea at all.

A recent cogent examination on how the benefits of big data being overblown comes from a recent article at Fortune, where contributor Ethan Rouen tries to bring down the big data hype a bit with the provocatively titled "Big Data won't solve your company's problems."

"But while the numbers don't lie, how people use them is extremely subjective. Quantitative analysis played a part in the financial crisis of 2007, after all, and companies that think a room full of analysts crunching numbers can solve their problems can do damage to not only their profits and losses but also to their corporate culture and employee well-being."

What's interesting about the article is that it doesn't use this theme to call the entire big data craze into question. The genie is out of the bottle, and right now the new arms race for businesses is how fast can they implement an end-to-end data strategy that includes high-end analytics and visualization?

What the article cautions readers to do (and here I agree) is not to rely on just data analysts ("quants" as they're known in the vernacular) to approach data analysis. But what articles like this miss is the "scientist" part of data scientist.

Scientists do two things over and over: hypothesize and test. And then test some more. And then a little more testing before lunch.

Now, the scientific method is not the be-all end-all for getting things right. Scientists can chase down the wrong path for years until someone figures out the problem. But it does lend an air of data skepticism that the subjects in this article are warning about. But the warning isn't necessary: if we can find enough real data scientists, the skepticism will already be in place.

The biggest problem for big data right now is finding the right sort of people that can perform this kind of work. It's not just about statistics and graphs… its about finding people passionate and knowledgeable enough about the subject matter to be able to find the answers they're looking for.

Are all the answers in the data? No. But there are more than enough to make the effort worth it.

Read more of Brian Proffitt's Zettatag and Open for Discussion blogs and follow the latest IT news at ITworld. Drop Brian a line or follow Brian on Twitter at @TheTechScribe. For the latest IT news, analysis and how-tos, follow ITworld on Twitter and Facebook.

ITWorld DealPost: The best in tech deals and discounts.
Shop Tech Products at Amazon