It’s Time to Think Differently About Mining Big Data

Everywhere we look today, big data is finally beginning to have an impact. Hospitals are using it to analyze patient populations, big pharma is using it to find new and better drugs faster, supply chain managers are squeezing more efficiency and timeliness out of their operations, retailers are connecting with customers in new and influential ways — the list is long and continues to grow daily.  While the headlines for big data are impressive, the use of traditional analytical methods could be leaving out hidden patterns.

The amount of data organizations that are amassing is growing and only continues to accelerate.  Traditional BI tools and reporting approaches are rapidly approaching their limitations in evaluating potentially billions upon billions of points of information.  These solutions are specifically designed to visualize and aggregate data to enable the user to make discoveries.  Whether through OLAP, reporting, or some sort of in-memory solution, all BI tools get you to the same end – some are just prettier or faster than others.

There are three ways how traditional BI has maxed its value with big data:

1. Analysis Only Provides Half the Story

BI gives you historical views of what has happened. It usually doesn’t show you why something happened or what will happen. Dashboards and reports are developed to give key decision makers information to make more informed choices, but have no ability to show the user how their decision will perform. Just knowing that something has happened doesn’t allow decision makers to have foresight into the outcome of a decision they are about to make. Prediction is the solution to stay ahead in business today, but it’s something that has eluded many due to the complexity and expense involved with implementing a predictive analytics strategy.

2. Big Data is Getting Too Big

While many organizations have come to know their own data over the past several years with the advent of user-driven BI technology, they are beginning to turn to external data sources to identify potential drivers of success and corner weaknesses. Bringing together internal and external data sources can exponentially grow the size and complexity involved with investigation or discovery. This breaks the traditional process of BI or data discovery; discovering patterns or concerns within these massive datasets is beyond what the human mind can investigate. Using BI to analyze our new world of data is kind of like getting to the top of Mt. Everest without a map; knowing the terrain you’ve already hiked on won’t help you navigate your way to the summit. This approach just doesn’t work because it doesn’t account for the constant changes in the mountain – the business.

3. The BI/Human Interaction Bias

Simply put, we’re all human. Our daily decisions are riddled with cognitive bias – some of which we are unaware.  Having knowledge of what has happened in the past still allows for an element of human bias; human bias is always brought into the reasoning process and can lead to missteps.  After dashboards and reports are consumed, there is always the potential for, “well, I know that’s not right,” or, “this is how we’ve always done it and it has worked.” Human bias not only leads down the path of skewed analysis of information provided in a BI tool or report, but can lead to missed opportunities.

With advances in both hardware and intelligent software technologies, the time is now to start thinking differently about how you are analyzing big data inside of your organization.

The Advent of Machine-Based Intelligence

Machine-based intelligence, or machine learning (ML), is something that has been around for decades but only recently has been a hot topic of conversation in the big data community.  Machine learning, more specifically, is a component of artificial intelligence that specifically refers to the ability of a computer to adjust and improve its own behavior based on experience. In other words, ML systems take in historical data points and use them to extract hidden patterns that are used to make increasingly better predictions about the future.

The use of machine learning technologies and techniques are showing up within the broader market. And this is for good reason – machines can exponentially increase the analytical capacity of any organization. Those billions upon billions of points of information can rapidly be processed and valuable connections made.

The use of ML in advanced and predictive analytics is where machines really begin to show their value. As a facilitator of information or a powerful data science tool to even the most astute data scientist, machine-learning technologies provide automation and continuous organic learning for one of the most tedious tasks when handling big data – data modeling and algorithm creation. By using machines to emulate the work of model and algorithm creation, organizations gain a powerful tool to rapidly mine big data properly while creating predictive intelligence for key business decision makers.

Operational reporting, dashboards, and traditional approaches to BI will always persist, and there are reasons behind utilizing these types of solutions. However, turning one of these traditional approaches loose on the problem of big data is like trying to find a needle in a haystack. As Moore’s Law continues to hold true and the ability to create more sophisticated software persists, using machine intelligence or machine learning to mine big data, automatically build predictive models and algorithms, and ultimately deliver better intelligence on what will happen–not just that something did happen–will drive greater adoption of these methods for mining big data.

About the author: Rob Patterson is vice president of corporate strategy at ColdLight, where he leads all go-to-market strategy, marketing, M&A, and strategic technology partnerships. Prior to ColdLight, Rob held senior marketing positions at Qlik. Prior to Qlik, Rob ran marketing programs in the Mid-Atlantic States District for Microsoft. Rob holds a degree in Food & Retail Marketing from Saint Joseph’s University.