June 18, 2013, 3:26 PM — Retailers place big bets on big data, digging to discover buying trends and preferences from masses of structured and unstructured data. Much of that data comes from outside their organization, in forms as diverse as research, reports, charts and even video files. However, while there's plenty of that data to preoccupy retail planners, that preoccupation often comes at the expense of gleaning intelligence from the data found in logs and other machine data produced by their own applications, websites, servers and supporting IT infrastructure components.
Machine data offers a wealth of information, but often goes untapped in the retailer's quest for external market research on buyer patterns, preferences and plans. And it's ironic that data from machine logs, while frequently overlooked, is always more time-critical than data from external sources. Machine logs that would warn of impending retail application downtime could save millions of dollars in lost revenues for a large online retailer during a holiday season.
In mid-2012, Target Stores lost, by one estimate, $464,000 in just 150 minutes of downtime caused by high server traffic; while that downtime could have been avoided by access to machine logs, external data would have been the electronic equivalent of a paperweight.
The big machine data
Logs and other machine data are the output of an organization's IT assets – essentially every application and device in the organization's IT infrastructure.
Machine data is one of the fastest growing components of big data, in part because it's generated by virtually every piece of IT hardware and every software application – servers, retail and related applications, and mobile devices – and sensors and input devices of all kinds. In fact, IDC forecasts that machine data will account for 40 percent of the total data generated in 2020, up from 11 percent in 2005.
By managing this data proactively instead of only when something goes wrong, organizations can help mitigate risk, ensure service availability and promote operational efficiency. It's an amount of data that many retailers are not prepared to struggle with, but tools such as log management systems are purpose-built to handle.
The anemic adoption of analytics for machine data may have several causes. In any case, 58 percent of retail industry professionals in a first-quarter, 2013, survey by Brick Meets Click stated they were concerned that their organizations were not using already available data.