Tip: Cascading, Open Source Java App Framework, Can Ease Big Data Hiring Challenge
According to Dun & Bradstreet CIO Walt Hauck, these service providers will be very good at helping you with first-order business problems, such as whether a marketing campaign is selling more products or what features customers are using in a particular product. However, he says, that's unlikely to happen with industry-specific questions that involve complex intensive analysis to find cause-and-effect relationships?
"You think an oil company's going to send [a service provider] a bunch of seismic data and say, 'Should we drill here?' Probably not," Hauck says.
Aside from the technology limitations, a shortage of data scientists drives this trend. The propeller heads who used to do regression and cohort analysis for the marketing department now find themselves in high demand and short supply. Great for them. Bad for you.
"There is a storm approaching on the big data talent front," according to the recent Big Data Executive Survey from New Vantage Partners. Writing in the foreword, Harvard Professor Thomas Davenport, co-founder and director of research at the International Institute for Analytics, notes that "70% [of respondents] say they plan to hire data scientists, but they already find this 'challenging' to 'extremely difficult,' and there is no reliable source of new talent in this category. It would seem to be a wise move to begin 'building' such talent as well as 'buying' it."
This new breed of data scientist needs not only to understand the new technologies around Big Data-the previously mentioned Hadoop, MapReduce and R as well as Pig, Hive, VMware Serengeti, NoSQL databases and so on&mash;but also has to understand how a business and its vertical market works.
Analysis: Does VMware Move Signal That Big Data Is Ready for Prime Time?
This a very rare breed at the moment, says Paul Barth, managing partner and founder of New Vantage. "To do these analyses, you still need really smart analysts. That's kind of a daunting resume."
Market for Big Data As a Service Small But Growing