Follow Datanami:
February 17, 2020

How is data growth affecting real-world enterprises? 4 key findings

Data volumes are expanding at an unprecedented rate. It’s very easy to get caught up in how many zettabytes of data we’ll all be producing by 2025, 2030, and beyond,  and how data and cloud computing affect us all on a global scale. But what do data growth and other issues mean for how you do business right now? The results of a recent Matillion and IDG Research survey are very illuminating.

Insight from 200 enterprise data professionals like you

Matillion and IDG Research recently conducted an IDG MarketPulse survey, “Optimizing Business Analytics by Transforming Data in the Cloud”. The survey polled more than 200 IT, data science, and data engineering professionals. The research exposes the challenges companies face and the strategies they use to prepare data for BI and analytics, with faster time-to-value for implementing analytics projects becoming the main driver for migrating to a cloud approach. Here are some of the key findings:

1. Data growth is hitting home for enterprises

Data professionals told us that, on average, data volumes are growing by 63 percent per month in their organizations. One in ten respondents told us that volumes are growing at 100 percent or more per month. Companies are at the receiving end of a firehose of data that keeps getting turned up higher. At that pace, it’s no wonder it’s so challenging to keep up, let alone turn that data into actionable insights.

2. That data is coming from hundreds (even thousands) of sources

We know that companies are collecting data from multiple data sources, including computers, smartphones, websites, social media networks, e-commerce platforms, and now IoT devices. But the actual number of data sources is astounding. According to the survey, the mean number of data sources per organization is 400 sources. More than 20 percent of companies surveyed were drawing from 1,000 or more data sources to feed BI and analytics systems.

3. Everyone will have data in the cloud within two years

Respondents said that all told, only 37 percent of organizational data is currently in cloud data warehouses. In addition, 35 percent of data is still in on-premises data warehouses. But those numbers will shift significantly in the near future: virtually all respondents (more than 99 percent) said they will migrate data to the cloud over the next two years. Why? Most people cited a faster time to value for implementing analytics projects. This would help businesses more effectively address common business problems.

4. Transforming data to make it analytics-ready is challenging for nearly all respondents

More than 90 percent of those surveyed said that it was challenging to some degree to make data available in a format usable for analytics. Of those, 57 percent said that transformation was highly challenging or extremely challenging.

Answers to another question may shed some light on why. Only 28 percent of respondents take an ELT approach: loading data into the cloud first, and then using cloud-optimized tools to transform it. And 37 percent said they are still manually coding data to get it into the needed format for analytics.

There are lots of takeaways from the survey. But a big one is that what we’re talking about on a universal scale is indeed affecting real companies across industries, and real data professionals. There’s absolutely a need for simple, fast, scalable, and cost-effective data transformation to help drive insights and innovation. Download the full report to find out how enterprises can take full advantage of cloud data warehouses and achieve rapid time to insight.