But there is another "elephant in the room" that you need to consider when looking to move to Hadoop-style data refineries: talent. Once companies have connectivity and their Hadoop clusters are integrated into existing business applications, you need the talent to make sense of it all. Hadoop cluster setup 101 and MapReduce jobs 101 aren't taught as part of the computer science curriculum at most major universities.
MORE: Hadoop wins over enterprise IT, spurs talent crunch
As the technology sector has begun to realize this lack of talent, they are responding accordingly: by breeding the data scientist. These scientists are typically some combination of computer scientist and mathematician -- and are seen as "data whisperers," always knowing what to ask the data in order to gain insights into the decisions that affects their companies.
As demand for these jobs grows, more focus at the university level will increase proficiency in this area and we will see a surge in data insights and innovation across all industries. Until then, the complete benefits of Hadoop simply cannot be realized.
The business value of Hadoop
Despite its shortcomings, Hadoop does offer huge potential for business value. As insights are gleaned from mining big data, more companies will seek to integrate Hadoop with existing applications. In fact, whether you know it or not, Hadoop has likely touched your daily life already.
In June this year, an article in The Wall Street Journal caused a huge stir by reporting that Mac users are more likely to spend more money on a hotel room than PC users. Given that, Orbitz searches pushed Mac users toward higher priced rooms.
This stat is based on giant data sets of information collected about the behavior of Mac users (750 terabytes of unstructured data, according to Tnooz) and is a great example of how companies are using Hadoop to create and store large databases of unstructured data, and analyze that data through a data refinery to make good business decisions through those analytics.
In this case, Orbitz stands to win if it can sell the right hotel to the right customer for a positive travel experience. By presenting pricier hotels to Mac users, who are stereotypically thought to have higher incomes, it hopes to boost its business based on huge amounts of data it has been collecting and analyzing for some time.
