Bad Data Pipelines Costing Companies Big, Fivetran Finds

Stop us if you’ve heard this one before: Overworked data engineer builds faulty data pipeline, which leads to bad data, which leads to a bad outcome. It may be the same old song, but it’s also the current state of affairs at many companies today, according to a report issued today by ETL provider Fivetran.

Fivetran’s new report, titled “The State of Data Management Report,” found that the average company is paying essentially wasting half a million dollars per year to employ data engineers to manually build and maintain data pipelines. Fivetran develops software that automates the development of pipelines that move data from a SaaS applications to data warehouses.

The company arrived at that number by figuring the average company employs about 12 data engineers at an average salary of nearly $100,000 per year. Based on the observation that data engineers are spending 44% of their time not working on advanced models and analytics, the rest is essentially wasted, the company says, with 85% reporting that bad data has led to bad decisions that have cost their companies money.

Not only are bad decisions being made from bad analytics based on bad data. But it’s also the fact that only 13% of survey respondents report that they can act upon new data within minutes or hours. For three-quarters of the survey-takers, it takes up to a week to prep the data for impactful analyses. That just doesn’t cut it, says Fivetran CEO George Fraser.

“What we’re seeing from this study is that data and analytics leaders are really struggling to keep up,” Fraser says in a press release. “It would be one thing if the processes companies used for manually building and managing pipelines were optimized, but 80% of those surveyed admit they have to rebuild data pipelines after deployment–from changing APIs, for example. For 39%, they say this happens often or all of the time.”

Source: Fivetran

The survey was conducted this fall by Wakefield Research. It included responses from 300 data and analytics leaders with at least a VP-level title at companies who work at companies with at least $100 million in revenue in the U.S., UK, Germany, and France.

The survey also found that 69% of data and analytics leaders report their business outcomes at their company “would be somewhat or significantly improved if their data team were able to contribute more to business decisions rather than manual pipeline management.” Nearly all the respondents (97%) say business outcomes would be improved “if their data team could spend more time devoted to the analytics behind data-driven business decisions,” the company says.

The poor state of data isn’t new. When you factor in the continued ubiquity of ETL and the widespread lack of investment in data management, it figures to be that way for the foreseeable future.

You can read more about Fivetran’s study .

Related Items:

Battle for Data Pros Heats Up as Burnout Builds

Digital Transformation Trumped Data During COVID: KPMG

Can We Stop Doing ETL Yet?