"Google has given us a map," he says. "We know where we're going. They started out publishing their GFS and MapReduce papers, which we quickly cloned in the Hadoop Project. Through the years, Google has produced a succession of publications that have in many ways inspired the open source stack. The Sawzall system was a precursor to Pig and Hive; BigTable directly inspired HBase, and so on. And I was very excited to see this year Google publish a paper called Spanner about a system that implements transactions in a distributed system-multitable transactions running on a database at a global scale. This is something that I think a lot of us didn't think we'd see anytime soon, and it really helps us to see that the sky's the limit for this platform."
Spanner, Cutting notes, is complicated technology and no one should expect to see it as part of Hadoop next spring. But it provides a route to the Holy Grail, he says. In the meantime, he points to Impala, a new database engine released by Cloudera at the conference this week, which can query datasets stored in HBase using SQL.
"Impala is a huge step down this path toward the Holy Grail," he says. "Now, no longer can you [only] do online puts and gets of values, you can do online queries interactively with Impala. And Impala follows some work from Google, again, that was published a few years ago, and it's very exciting. It's a fundamental new capability in this platform that I think is a tremendously valuable step on its own and will help you build more and better applications on this platform. But also I think it helps to make this point, that this platform isn't a niche. It isn't a one-point technology. It's a general purpose platform."
We know where we're going with it," Cutting says, "and moreover we know how to get there in many cases. So I encourage you to be comfortable adopting it now and know that you can expect more in it tomorrow. We're going to keep this thing advancing."
Thor Olavsrud covers IT Security, Big Data, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline and on Facebook. Email Thor at email@example.com
Read more about business intelligence (bi) in CIO's Business Intelligence (BI) Drilldown.