"There [will be] more career opportunities in the future for this type of strategic analysis," says Kelley, who has seen the business intelligence analyst job change into a data scientist position in the last 18 months. "We've always used information but not to this level. With the amount of data companies are capturing on everything and everybody it's just amazing what can be done with that."
Colleges have realized the need to train people for those careers and are developing degree and certification programs targeting undergraduate and graduate students as well as IT professionals. To address immediate data-science staffing needs, which include technical and business roles, companies have adopted assorted tactics.
To handle the more than 100TB of data processed each week by BrightEdge, a San Mateo, California, startup that helps companies manage their search engine rankings, CEO and founder Jim Yu wants data workers who grasp the entire scope of big data processing.
People know how to query databases, but there is "an extra layer of understanding" when handling large data sets, which at BrightEdge includes tracking data on more than 150 billion URLs. Experience working with traditional SQL relational databases helps, but big data's scale requires a different processing mindset, he said.
"There's a nuanced leap there when you move into this big data environment," Yu says. "You're really looking at the optimal configuration of taking these massive processing jobs and figuring how do you distribute this load on servers that are much less monolithic and much more distributed."
In addition to database knowledge, Yu notes that strong backgrounds in computer science, algorithms and OSes are helpful bases to a BrightEdge data science career.
"If they have a good foundation in that, then you pair that up with a [training] program that allows them to understand how to translate into this new architecture," he says. This buddy system, which matches workers who have worked on the big data stack with people who are learning the system, leads to knowledge sharing, he said.
This method also helps people new to extreme data crunching learn which data processing jobs call for big data technology and when to use traditional relational databases, Yu says.
"With big data, one of the advantages is the scale of what you can do," he explains. "But it also means you don't have the same speed of development from having the really simple, flexible standardized SQL language that you can apply to the data set. There are tradeoffs that you're making. It's important for the technology staff to have a good appreciation of that."