One of the pain points experienced by just about any organization that seeks to deploy Hadoop is the shortage of Hadoop skills among the IT staff. Engineers and developers with Hadoop chops are difficult to come by. Gartner's Adrian is quick to note that HDInsight in either flavor won't eliminate that issue, but it will allow more people in the organization to benefit from big data faster.
"The shortage of skills continues to be a major impediment to adoption," Adrian says. "Microsoft's entry does not relieve the shortage of experienced Hadoop staff, but it does amplify their ability to deliver their solutions to a broad audience when their key foundation work has been done."
"It opens the world of data outside traditional data stores to commonly used business tools in a way that will accelerate adoption by today's business analysts-but not without the infrastructure creation that will still depend on the skills of technical professionals using Hadoop technologies," he adds.
Seamlessly Move Big Data Between Cloud and On-Premise
One immediate benefit, he notes, is the ability to rapidly prototype in the cloud and then seamlessly bring the solution on-premise using the common platform offered by Windows Azure HDInsight Service and Microsoft HDInsight Server for Windows.
"Early experimentation and testing in the cloud eliminates early capital expenditures that can be a barrier to adoption," Adrian says. "For example, Amazon's AWS has been used for millions of Elastic MapReduce jobs, but those experiments stayed in the cloud. The connection of cloud to on-premise systems offers the promise of more rapid time to value-and Microsoft has established a lead in connecting to its widely used stack."
Microsoft's Leland notes the capabilities provided by the two flavors of HDInsight also allow customers to run most of their big data analytics on-premise but cloudburst to Azure when they need the additional compute.
"Ultimately it's an ideal scenario," he says. "You can manage your infrastructure cost, but really have unlimited scale when you need it. You can spin up a cluster in less than 10 minutes [in Azure]. Having that kind of additional capacity, that quickly, on demand, that simply, is certainly going to change the game."
Microsoft Committed to Open Source
For its part, Hortonworks is working closely with Microsoft on its big data initiatives.
"This is a joint engineering effort," explains John Kreisa, vice president of marketing for Hortonworks. "This is an engineering relationship in which our engineers have been working with Microsoft engineers in porting Hadoop, which is traditional Linux infrastructure, over to Windows."