That's big data. If you can learn to drink from the fire hose, it can provide the sort of intelligence and actionable insight that business leaders dream about. On the security front it can help you protect your organization from advanced persistent threat (APT) attacks and malware by providing visibility into what's happening in your network, and it can give forensics a huge boost as well. It can also lead to tremendous gains in operational efficiency, from optimizing your servers to optimizing your supply chain management. It can even help you get a handle on compliance issues.
But if you don't have the tools to manage and perform analytics on that never-ending flood of data, it's essentially garbage.
Khera says one of the keys to getting big data under control is log management that consolidates and centralizes logs from across an organization-including logs from web applications, middleware, custom backend applications and databases -- with an indexed storage repository and common user interface. To make sense of the data requires the ability to normalize it, correlate it, report on it and send actionable alerts.
Earlier this month, LogLogic commissioned IANS, founded as the Institute for Applied Network Security, to perform an Information Security Investment Analysis (ISIA) of its log data management and compliance products.
After interviewing a number of LogLogic customers dealing with big data issues, IANS said, "The major differentiator with big data log management is the sheer size of the amount of log information. Trying to recreate an event after the fact is no simple matter if only a few devices are available. Imagine looking across thousands of devices and through petabytes of data without having an easy-to-use UI or an indexed storage repository for rapid response. Big data is characterized not just by size but also speed. Searching through massive amounts of data takes time if it's not properly indexed. If critical information about unauthorized access or other activity is not available because it hasn't been indexed, the results of a search will be inconclusive. Thus a bid data management solution must be able to keep up with the onslaught of new messages. This is even more important when it comes to alerting. If the indexing is taking too long, critical alert messages are delayed causing unacceptable latency in response times."
For now, though, only 54% of respondents said they use a log management solution to manage their log data. Many use syslogs and spreadsheets to manage their logs, according to the survey, and 33% do nothing at all.