It's also crucial to acquire experience with processing large amounts of information, Fuller says. "You need to handle data sets that are hundreds of millions of records, detect glitches in that data and know which statistical tools to apply," he says.
Applications that are appearing on the market from vendors such as Fremont, Calif.-based Accrue Software Inc. and Lanham, Md.-based Group 1 require less knowledge of statistics and programming. "More and more, data mining technologies are becoming embedded in vertical applications," says Judson Groshong, vice president of marketing at Accrue.
"We have hidden the details of the actual algorithms so that the only things users see are the business parameters," says Groshong. The applications are simple enough for a businessperson to use, but a technologist still needs to prepare the data and ensure its accuracy, he says.
Good Data Miners
But that won't endanger a data miner's job. "The thing that makes good data miners better than mediocre ones is something that is hard to teach and impossible to automate: a good intuition for what variables are likely to be useful and a feel for how to coax information out of data," Berry says. Although tools can automate the model-building process, "only the human knows to replace a ZIP code with characteristics of that ZIP code, such as median income and ratio of renters to owners."
People who work in data mining say that despite the many challenges they face, the rewards are great. James says that for him, the greatest challenge -- and reward -- is the unglamorous side: getting data out of the warehouse or legacy systems and validating it. "There's nothing better than coming out of a meeting knowing you've presented results that are meaningful to the audience and are actionable," he says. "It's not uncommon for us to provide results that can translate immediately into millions of dollars of saved costs."