Deep into big data

By Divina Paredes, CIO New Zealand |  Big Data, Analytics, big data

Democratisation of data inside a company to make a business agile is important, says Kee Siong Ng, senior data scientist at EMC's Greenplum.Traditionally, he says, there is a conflict over how to classify data mining -- is it an information technology application or part of the wider business. "You see it moving back and forth," said Kee Siong in his keynote address at the 2012 SUNZ Conference in Wellington last week. "One thing you must do is try to formalise that relationship, and really emphasise this collaborative environment," he says. "You might have the infrastructure being managed by IT, but the data side is really owned by the business." This division, he says, is a major problem in a lot of organisations, except in internet companies like Yahoo and Google. "Even in places where they have done data mining in 10 to 15 years, they still have that wall."His advice to enterprises: "Put all your data to work, have a data strategy, first invest in people, then technology," he says. "It is all about efficient, agile analytics."Veering from traditionHe says a typical scenario in organisations is that critical data goes to a large data warehouse, which is controlled by IT. In this system, you can do "shallow type reporting", and then there will also be "shadow systems" built across the organisations to support new data sources.Users will have to export data from the data warehouse and combine this with data from the "shadow systems" to do analysis. Most of the time, they are also working on small samples so the analysis is not complete. He describes a new analysis practice for big data which he calls "MAD" for magnetic, agile and deep. The goal is to build models using all available data, he says. Every time you get new data sources, you don't have to worry about how it fits with the current scheme, you just put it inside the analytics warehouse in its raw form. Agile allows analysts to work faster, and makes it easier for them to push back the results for deployment. Deep analysis allows data mining in large datasets.High performance analytics in actionJames Foster, chief technology strategist for SAS ANZ, says there is misconception that big data refers to companies like Amazon and Google. "It is not just about volume," he says. "It is not how large your data is that you can actually manage. [But] What is relevant for your organisations to make decisions and using the right information?"High performance analytics in the context of big data is turning that data into information and insight, he says. "There is a difference between the information they are currently using and they could be using to drive value."Across industries, he says, there are specific areas with lots of challenges around big data. In insurance, for instance, the business issues are around telematics, claims analytics, ratemaking and catastrophic modelling.


Originally published on CIO New Zealand |  Click here to read the original story.
Join us:
Facebook

Twitter

Pinterest

Tumblr

LinkedIn

Google+

Answers - Powered by ITworld

Join us:
Facebook

Twitter

Pinterest

Tumblr

LinkedIn

Google+

Ask a Question