February 26, 2012, 7:29 AM — Big data gets a lot of buzz these days and organizations are increasingly concerned about the problem of managing it, but many don't really understand what big data is. Nor do they have the tools in place to effectively manage much of the data already at their disposal, says Mandeep Khera, chief marketing officer of LogLogic, which specializes in a scalable log and security intelligence platform (LSIP) for the enterprise and cloud.
"Most of them are concerned about big data, yet they don't understand what it means," Khera says. "Because there's been so much said about big data, there's no clear definition and everyone is confused."
A new survey conducted by LogLogic in conjunction with IT security research consultancy Echelon One finds that 49% of organizations are somewhat or very concerned about managing big data, but 38% don't understand what big data is and a further 27% say they have a partial understanding. Additionally, the survey found that 59% of organizations lack the tools required to manage data from their IT systems, instead turning to separate and disparate systems or even spreadsheets.
"We know that data is important from a lot of different perspectives: security, IT operations, compliance," Khera says. "Companies need to be managing data much more effectively so they can make more intelligent decisions."
The global survey was based on the responses of 207 individuals at director level and above in a variety of industries, including manufacturing, education, government, finance, healthcare, transportation, media and publishing and others.
"Big data is about many terabytes of unstructured data," Khera explains. "Information is power, and big data, if managed properly, can provide a ton of insight to help deal with security, operational and compliance issues. Organizations of every size are collecting more data from a variety of sources within the enterprise and cloud infrastructures, and many organizations are not using the right tools and processes to manage these data. If this pattern continues, we will see enterprises falling further behind, unable to derive actionable insights which can help organizations make intelligent decisions."
Most respondents to the survey -- 62% -- said they already manage more than one terabyte of data. But more is coming. The volume of data in the world is increasing at a nearly incomprehensible rate. IBM says we create 2.5 quintillion bytes of data every day. And perhaps more astonishing, 90% of the data in the world today was created in the past two years according to Big Blue. The data is coming from sensors, transaction records, images and videos, social media posts, logs and all sorts of other sources.
That's big data. If you can learn to drink from the fire hose, it can provide the sort of intelligence and actionable insight that business leaders dream about. On the security front it can help you protect your organization from advanced persistent threat (APT) attacks and malware by providing visibility into what's happening in your network, and it can give forensics a huge boost as well. It can also lead to tremendous gains in operational efficiency, from optimizing your servers to optimizing your supply chain management. It can even help you get a handle on compliance issues.
But if you don't have the tools to manage and perform analytics on that never-ending flood of data, it's essentially garbage.
Khera says one of the keys to getting big data under control is log management that consolidates and centralizes logs from across an organization-including logs from web applications, middleware, custom backend applications and databases -- with an indexed storage repository and common user interface. To make sense of the data requires the ability to normalize it, correlate it, report on it and send actionable alerts.
Earlier this month, LogLogic commissioned IANS, founded as the Institute for Applied Network Security, to perform an Information Security Investment Analysis (ISIA) of its log data management and compliance products.
