What Is An Insight Engine? And Other Questions

(Photon photo/Shutterstock)

We collectively spend huge chunks of our lives wading through data in search of information that’s useful. In fact, while there are many different approaches to it, that is what the big data game is all about. Now an emerging class of systems dubbed “insight engines” promise to automate that information retrieval task, to one degree or another.

Insight engines are a relatively new class of big data product that emerged recently from the world of search (which also gave us Hadoop). Gartner is at the forefront of the trend, and recently issued a Magic Quadrant for Insight Engines to replace the same for enterprise search.

To get the low down on insight engines, we caught up with Daniel Fallmann, the CEO and founder of Mindbreeze, an Austrian software company that was featured in that recent Magic Quadrant. Here’s an edited version of an email Q&A with the CEO:

Datanami: First of all, how do you define insight engines?  What are they composed of, and how is it different than an enterprise search engine?

Daniel Fallmann: The term “insight engines” was essentially invented by Gartner. Their definition of an insight engine reflects our view as well. To be more concrete, this means: Insight engines augment search technology with artificial intelligence to deliver insights — in context and using various modalities — derived from the full range of enterprise content and data.

Daniel Fallmann is the CEO and founder of MindBreeze

Accordingly, behind the term “insight engine” lies an intelligent solution that makes it possible for information to be found in a way that is resource-efficient and makes that information available to the user in the right context for the respective business case. In doing so, these systems use methods of artificial intelligence to capture and collect existing corporate knowledge, extract the information, and show correlations between the individual pieces of data in order to convey a comprehensive overall picture.

The applied artificial intelligence and the effective context-aware presentation of information make the difference between insight engines and traditional search technologies: Insight Engines use machine and deep learning to extract information and bundle enterprise knowledge and make this a self-learning process. Based on previous events and user behavior analysis, the technology learns to categorize information to provide a personalized comprehensive picture to each user.

Using natural language processing (NLP) and natural language question answering (NLQA), search queries can be delivered in natural language and processed directly. These intelligent technologies can analyze and understand structured metadata as well as text content, and use this to correctly determine what the user needs. While NLP deals with human language, NLQA enables the linguistic interpretation of search queries. These technologies can identify the specific needs of the users and tailor the search results to correspond perfectly to those needs. The objective is to pinpoint the user’s needs and to match the search results in the respective context, so that instead of an endless list of search hits, the user only receives the results that actually correspond to the searched term − augmented by context-specific additional information generated by semantic analyses.

DN: Why is it sometimes necessary to create custom insight engines? What benefits did they bring?

DF: Of course, customer needs and expectations have evolved radically in recent years and different customers from different industries clearly have different needs. Nevertheless, their departments have quite similar requirements across the various industries.

The old and traditional way of building a custom insight engines was to start a project with specifications, development, and roll-out. Modern products are developed to work like a customer self-service shop. The departments know their information requirements (such as having a 360° view of a product or a customer) and are able to compose their own search apps and dashboards and relate information in a way that is context-aware and intelligent. This makes a modern, out-of-the-box insight engine as flexible as a traditional custom insight engine, but way faster, 100% focused on the end-user’s information needs, and overall a lot easier to use on a day-to-day basis, even for large-scale enterprises.

DN: Why is the custom approach for insight engines no longer feasible?

DF: Almost 10 years ago, we already realized that it is more sensible to focus on product innovations rather than on long-running project customization. At Mindbreeze, we wanted to develop an effective and flexible tool that is able to support employees from all industries and departments and one that companies can easily implement without the need for an expensive and time-consuming implementation project. Our focus was to use the best practices from several industries and apply them to others, which worked out very well as their use cases and needs are quite comparable on a departmental level, such as for customer service departments.

DN: Can out-of-the-box insight engines deliver everything that a custom insight engine can?  What compromises must they make to go the out-of-the-box route?

DF: An out-of-the-box product offers the huge advantage that it is immediately ready to use. So obviously an out-of-the-box insight engine can deliver everything that a custom insight engine can, but a lot faster. To be able to do that, we have developed a microservice architecture that is deployed as an appliance or in the cloud and offers so-called extension points to hook any customization into a microservice that is needed to fulfill a customer’s needs. Those extensions can be easily implemented in the company without the need for expensive long-term project plans.

(Panchenko Vladimir/Shutterstock)

Using connectors for the different data sources such as network drives, SAP, Microsoft SharePoint and numerous ECM systems, all corporate data can be easily and efficiently integrated into the system, making this an effective solution for big and small companies from a wide range of industries.

DN: How much customization is possible with out-of-the-box insight engines?  What elements can be “tuned” or configured without the time and expense of a full-bore custom development effort?

DF: An out-of-the-box insight engine does not mean that you can’t customize it. Depending on the knowledge of the IT-staff, almost any adaption is possible.

There are solutions that provide a nice combination of both pre-built search apps from our marketplace and tools for customization. The apps range from editors that contain a selection of design elements, or widgets, such as filters − which can be easily combined just the way you want them without any prior programming knowledge − to advanced tools that allow high flexibility in customization.

DN: How popular is the out-of-the-box approach to insight engines today?  How quickly is that changing in comparison to the custom route?

DF: The trend is definitely towards out-of-the-box and subscription-based services that customers can use as they need them. In this extremely fast-moving and dynamic age of digitalization, it is more important than ever to be able to react flexibly and quickly. Companies do not have the time – and certainly not the necessary resources – for expensive and time-consuming projects. Our recommendation is to use products and services that already serve your needs instead of starting a long-term project to build something on your own.

Related Items:

Intelligent Search Delivers Big Benefits

Rethinking Enterprise Search for the Big Data Age