Zen and the art of enterprise search

What's the similarity between meditation and business? Simple: In both cases, the answers to the big questions are usually found within. And with widespread deployment of business intelligence and data warehousing solutions among corporates, that's never been truer than it is today.

When knowledge workers want to know something, they are looking inward, using search techniques to seek out clues that are scattered across the entire corporate IT resource--from the data submerged in silos including ERP, ordering and financial systems, intranets, e-mail and Web servers, and users' own workstations.

This involves sifting through multiple sources and linking together the pieces of information to give a starting point for decisions. The past three years have seen a real focus on enabling knowledge workers to do this more efficiently, because without effective companywide search techniques, users end up wasting time as they hunt across multiple systems for a piece of information. In turn, companies suffer too.

Worse still, has the same information already been searched for and found by another person--only for someone else to have to repeat the whole process? A recent U.S. study asserted that knowledge workers spend more than twice as much time re-creating content as they spend creating new content.

What are you searching for?

This explains the growing popularity of enterprise search techniques with larger companies. From the user's point of view, enterprise search works in a similar way to using Internet search engines such as Yahoo. Using a simple, logical interface and the ability to search using keywords typed in everyday language, people get search results organized by ranking that they can further refine.

Much has been made of enterprise search's ability to locate and sort unstructured information--that is, data held in Word and PDF documents on servers and users' desktops that isn't indexed, tagged or archived for easy location. U.S. and European analysts agree that about 80 percent of information within businesses is unstructured, and so the majority of tools have focused on this aspect of searching.

But what of the other 20 percent of information? That's the data that lies in business intelligence (BI) systems, in data warehouses and core systems such as ERP. This is the data that has the real business answers, because it includes financial transactions, point-of-sale information, sales and CRM records, and other business events.

The problem is that many enterprise search tools fall short when it comes to searching this structured data--which means users don't get access to the most important information to help make their decisions.

Super structure

No wonder, then, that the ability to search structured data is causing the biggest stir among corporate information managers. With data volumes growing by 200 percent a year, knowledge workers are spending around 30 percent of their time trying to extract useful information from these swelling data silos.

What's more, conventional approaches to information discovery and access on structured data are running out of steam; traditional proprietary applications are often too complex for people to use without expertise and specific training. Also, software licenses are too expensive for wide-scale deployment to all the knowledge workers who may need access.

Enterprise search technology offers a solution to much of the cost and complexity of accessing structured BI systems--if the right search tool is chosen. So how do you ensure that you choose the search solution that delivers real, ongoing business ROI? Here's a checklist of the main points to observe:

Structured data matters

First, check the ability of the enterprise search tool to search structured data sources such as ERP, sales systems, CRM systems and customer databases.

Look for solutions that draw on a BI background, enabling users of all levels of expertise to search databases and other structured data sources, using the same natural-language keyword search facilities and interface that they would use for document searches.

This gives users deeper access to key business information, without the need for training, additional software licenses for existing systems or interfaces to BI tools--giving direct savings in both costs and time. It also obviates the costs and effort of deploying data cleansing and conversion, which is the common obstacle to querying of data originated across multiple application silos.
So ensure that the solution you choose is able to index and search structured data sources as well as unstructured sources, to get maximum value.

Results presentation

Second, the ability to perform a search itself is not the answer; it's all about the results. Users want to be able to explore, map, correlate and analyze those results so that the answers to their queries make real sense. And some enterprise search solutions go much further than others in terms of presenting search content and giving users a real insight into their results.

Typically, most search solutions simply return search results as a list of text matches--much like their Internet search cousins. This does little to show the user how closely the result matches their search, or the relevancy of the data.

What's more, if a user conducts multiple related searches, working with lists of results makes correlation and analysis time-consuming and unwieldy. Look for a solution that goes beyond simple lists of results, giving users a range of powerful presentation-layer functions.

These enable overlaying of results of multiple related searches--allowing users to explore commonalities between groups or search results, building an in-depth picture of results from seemingly unconnected data.

What's more, they can display the results of these searches using visual techniques such as Venn diagrams, giving users a quick and intuitive grasp of how search results from individual queries relate to each other. For example, a user in the insurance sector can overlay multiple searches across types of motor claim, geographic region, sex of claimant and so on, to investigate concurrencies and possible patterns.

It should be possible for these searches to be conducted and updated in real-time, so that when additional data is overlaid, the new search can also update earlier searches to give an up-to-the-minute snapshot of information and its relationships.

Choosing the solution that combines in-depth search capability across all data sources with rich presentation functions will help companies to get the most value from enterprise search--and by extension, from their existing business systems.

Richard Lewis is technical director of BI specialist Ardentia, which can be found at www.ardentia.co.uk.