Follow Datanami:
October 1, 2020

Pandemic Driving ‘Back to Basics’ in Big Data, Study Suggests


During a financial crisis or natural disaster, stock traders frequently abandon risky equities and instead move to safer bets, like utilities, precious metals, and consumer goods. We may be seeing the same sort of flight to safety in the big data world as a result of all the disruption caused by the COVID-19 pandemic, a recent study suggests.

In its State of Data Culture Report, which Alation officially unveiled yesterday, 300 C-level executives, mid-level execs, and managers at mid-to-large companies (2,500 employee minimum) were asked a series of questions about their data and analytics investments. The survey revealed some interesting data tidbits, including the existence of a “data culture disconnect” in how organizations perceive themselves versus their actual capabilities.

Part of the survey had to do with how the COVID-19 pandemic had impacted their big data projects. When survey-takers were asked what tools they are using more during the start of the pandemic, business intelligence and data visualization tools led the list, with 47% and 43% of respondents indicating their usage was up, respectively, outpacing increases in AI, machine learning, forecasting, and dashboards.

Alation co-founder and Chief Data and Analytics Officer Aaron Kalb has some ideas why this may be the case.

“One thing that I was really interested and excited to see was how COVID has changed behaviors in organizations,” says Kalb, who spearheaded the survey. “In particular, if you look at the distribution of responses, there are folks who talk about new machine learning, new AI, and new fancy data science stuff. They’re excited about that stuff they’re doing internally, and I love that.

Source: Alation State of Data Culture report

“But the thing that’s seen the biggest increase actually is BI and visualization and cataloging, which you could describe as a back to basics to approach,” he continues. “The idea is, when the world is super uncertain, instead of this attempt to bet on some crazy algorithm…let’s get really good at sourcing the right data and looking at the right data and doing the right thing.”

The survey also found that 58% of C-level executives say they are relying more on data to make decisions than they were before COVID-19. That was higher than other cohorts identified in the survey, including mid-level executives and management.

The COVID-19 pandemic has created massive amounts of stress for billions of people around the world, and continues to wreak havoc on human institutions, with no end in sight. We are all trying to come to grips with what is happening, and so it’s natural that we turn to data to help us make sense of it. Frontline healthcare practitioners are also turning to data, although the results are not always good.

The future of big data undeniably involves heavy doses of machine learning and artificial intelligence, particularly for high-volume and complex data sets. This is what Datanami focuses on. But in many cases, the data that we need to get answers from are not always large and not always complex. In fact, it’s not always clean, and much of it could probably fit into Excel, which continues to be the world’s number one data analysis tool.

There are many tools that are better and easier to use than Excel, and most of them now are beginning to incorporate machine learning technologies (according to Gartner), which makes them even better and easier to use. But when it comes to spotting patterns in human-generated data, there is often no substitute for human eyes and human brains.

There is good reason to believe that Kalb’s hypothesis is correct, as there are still big gaps in distribution of data and AI capabilities across the land. Yes, some firms have graduated from BI and visualization tools and are now looking to up their games by automating the detections of anomalies and finding patterns using machine learning and AI.

But everybody isn’t there yet, and for many folks, simply having a way to coral data with a catalog and get eyeballs on it with a BI or viz tool (particularly the ones that have ML embedded in them) can deliver a sizable return on investment. Perhaps it just took a calamity like COVID-19 to cut through the hype and show what matters most.

Related Items:

Data Culture ‘Disconnect’ Identified in New Index

How the Lack of Good Data Is Hampering the COVID-19 Response

ML and BI Are Coming Together, Gartner Says