DataOps Hiring Surges Thanks to ML, Real-Time Streaming

(Rawpixel.com/Shutterstock)

Nearly three-quarters of companies plan to hire a data operations (DataOps) professional in 2018, according to a new survey commissioned by Nexla, which says investment and interest in machine learning, artificial intelligence, advanced and real-time analytics is driving the demand.

There’s no doubt about the growing importance that data represents to many companies. The potential to use machine learning, AI, and streaming analytics to make better decisions and provide superior customers service is seen as a major difference maker by those in the C-suite.

However, getting all this data landed, transformed, cleaned, and ready for analysis isn’t easy, particularly when teams largely rely on manual methods for various data ingestion, transformation, and management tasks, which is pressing CIOs to look for a better data approach.

Data format consistency was the number-one challenge cited by data pros, according to Nexla’s survey

DataOps is that new approach, and companies are beginning to figure that out, which is why 73% of companies are looking to hire at least one DataOps professional this year, according to Nexla’s  Definitive Data Operations Report 2018.

Nexla, which develops a DataOps platform, says DataOps “takes care of the grunt work typically placed on IT or data engineers. This includes integrating with data sources, performing transformations, converting data formats, and writing or delivering data to its required destination.” Monitoring, governing, and securing the data also fall under the rubric of DataOps responsibilities.

Nexla found that 85% of companies have teams working on ML or AI projects. That’s up from 70% who said they were working on AI and ML last year, accusing to Nexla. More than half of respondents said they’re using real-time streaming data today, and nearly two-thirds say their use of real-time streaming will increase.

Nexla says the average company lands 2.7TB of new data per day, which isn’t surprising. However, the diversity of data sources is also increasing, and 85% of companies ingest data from third parties, and more than half (54%) are ingesting data from more than 10 partners.

The growing complexity of data and data systems is putting stress on engineers, scientists, and IT staff. According to Nexla, 50% of companies say there are not enough backend resources to complete their tasks. Companies with more front-end folks (i.e. data analysts, data scientists, and business users) were more prone likely to say the backend resources (i.e. software and data engineers) were strained, Nexla says.

(Source: Nexla’s  Definitive Data Operations Report 2018)

The average engineer spends 18% of their time troubleshooting problems, which they identified as their least enjoyable activity. Among data scientists, 44% cited data clean up and preparation  as their least enjoyable task.

Both of these tasks could be automated to some extent with DataOps. Other parts of the data value chain that could benefit from a dash of automation include analysis, integration, pipeline building, data modeling, and managerial activities.

“DataOps is as much about people as it is about tools and processes,” states Nexla co-founder and CEO Saket Saurabh in a press release. “To really drive value from machine learning, AI, and advanced analytics, data teams need to stop troubleshooting and start automating.”

Nexla’s report can be accessed at www.nexla.com/definitive-data-operations-report-2018/.

Related Items:

MapR Takes Aim at DataOps with Platform 6.0

Data Ops Upgrade Targets App Performance

DataOps Dominates Strata Startup Winners