Observability and AIOps Tools Rise with Big MELT Data

(Virrage Images/Shutterstock)

The hottest space in big data at the moment arguably exists between the two pillars of AIOps and observability, where a crush of metric, event, log, and trace (MELT) data threatens to overwhelm IT professionals. Not surprisingly, we’re seeing a rush of vendors to satisfy the market need, field by investments from venture capitalists and private equity firms.

We’re in the midst of a perfect storm of sorts when it comes to the world of machine generated data created by our digital world. Applications are getting bigger (i.e. becoming more distributed and linked via microservices) while they’re simultaneously getting more complex (i.e. container-based and probabilistic via AI). Add to that increasing customer demands for a smooth 24x7x365 experience, the need for rapid updates via modern CI/CD pipelines, and the continued evolution of The Great Cloud Migration, and we have a situation where the IT professionals are becoming overwhelmed by MELT data.

That’s where observability and AIOps comes in. By collecting and analyzing the MELT data, observability tools give IT pros the ability to at least monitor all this data and regain insight into what is happening in their systems, which is something that traditional monitoring tools are unable to do. And when it comes time to move beyond monitoring and into managing this morass of next-gen digital goodness, AIOps tools give IT pros the machine learning-powered advantage they need to effect change.

In its 2021 State of Observability report, VMware found that, while only 16% of firms have adopted observability tools, the field is “rapidly gaining traction.” The number one reason: “A fragmented monitoring landscape across an organization results in limited insights into the performance of cloud applications.”

Four essential types of telemetry data (Image courtesy Rajesh Kumar and DevOps School) 

VMware says observability tools, such as its Tanzu offering, give DevOps and site reliability engineering (SRE) teams the capability to keep up with these rapidly changing environments. Among organizations that have adopted observability, 92% say they are “necessary or very valuable,” VMware’s survey says.

Many of the big existing players in log management and systems monitoring have jumped on the observability train. Splunk–which turns 20 years old this year and, with an $18-billion market cap, is no longer a plucky startup–touts its full-stack observability platform. Cisco, which paid $3.7 billion for AppDynamics five years ago, has is also touting its full-stack observability capabilities.

Startups are also getting into the observability game, often with the backing of outside capital. Last fall, San Francisco-based Honeycomb raised $50 million in a Series C round of funding led by Insight Partners, bringing the company’s total capital raised to nearly $97 million.

“Observability is succeeding APM because it enables teams to quickly find critical issues in their cloud-native, microservices-based apps,” the company says. “Modern microservice architectures increase velocity and scale but also add painful complexity and unpredictability, which legacy APM tools fail to debug because they were built to examine simpler monolithic applications in predictable environments.”

Dynatrace is another traditional APM vendor that is heeding the siren call of observability. In a recent survey, the company found that 61% of organizations say observability “blind spots” in their multi-cloud environments are becoming a greater risk to digital transformation as teams find themselves without an easy way to monitor their infrastructure end-to-end.

Observability data threatens to overwhelm traditional monitoring tools (Chim/Shutterstock)

“Multi-cloud strategies have become critical to keeping up with the rapidly accelerating pace of digital transformation,” Dynatrace founder and CTO Bernd Greifeneder stated, “but teams are struggling to manage the complexity that these environments bring.”

There are many plucky startups aiming at the big observability dogs, such as Hydrolix. The Portland, Oregon, company has devised what it considers a superior compression mechanism which, combined with the ability to run in the cloud and store data in S3, allows customers to keep bigger stores of observability data than other providers. The company has raised $10 million and is backed by Wing Venture Capital.

Cribl is another benefactor of the rush into observability tools, which can sometimes create a data storage and processing crunch in its own right.

“We sit between sources and destinations of this observability and monitoring data,” says Nick Heudecker, senior director of market strategy and competitive intelligence for Cribl. “Basically we’re kind of restoring a lot of choice and control over what people can do with all of this observability data.”

The San Francisco company, which has raised more than $252 million through three rounds of financing, is seeing increased demand for its data observability pipeline software. Earlier this month, the company announced that it tripled its customer base and its revenues in 2021, and demand continues to look strong in the first couple of months of 2022.

Meanwhile, observability’s little brother, AIOps is also enjoying a moment in the sun. Interest in AIOps, which is a term Gartner coined a few years ago to refer to AI-driven IT operations, is growing for many of the same reasons that observability is growing: the lack of control over an increasingly complex digital infrastructure.

AIOps gives IT pros a leg up on managing modern systems (ART-STOCK-CREATIVE/Shutterstock)

The key differentiator between observability and AIOps is what you do with the data. In observability, it’s often enough to just have visibility into what’s going on. With AIOps, IT pros take the next step in automating the management of increasingly complex systems. In this case, it’s AI providing the helping hand. (The line separating the two may be murky at times, which invariably is the case with fast-moving IT terminology.)

Virtana raised $73 million in January to help grow its AIOps solution for hybrid cloud applications. The money, which came from Atalaya Capital Management, Elm Park Capital Management, HighBar Partners, and Benhamou Global Ventures, is aimed at helping to capture a share of what they claim is a $30 billion market for IT operations management software.

In January, BigPanda raised $190 million at a $1.2 billion valuation to help it grow its AIOps platform, which is designed to help IT operations teams streamline the collection and analysis of event data to keep their systems running smoothly.

According to BigPanda’s 2021 AIOps Benchmark Report, more than 90% of organizations are already investing in AIOps or plan to soon. While only 16% of survey-takers say they’re already using AIOps, these early adopters say they’re finding the software makes a “significant operational impact” on their organization.

The funding round was headed by Insight Partners, a private equity company based in New York City. Insight Partner Managing Director Lonne Jaffe, who was the CEO of Syncsort a decade ago, sees big growth in AIOps in the years to come, and is placing his bets on BigPanda.

“…BigPanda is now enjoying powerful network effects as they sit in between the ‘pandamonium’ of the upstream IT data, on one side, and the downstream collaboration systems, on the other side,” Jaffe stated in a press release. “BigPanda’s extraordinary momentum is a testament to its product differentiation, customer success maturity and skilled leadership, and we couldn’t be more excited to support the company’s growth into the future.”

Another big round went to ScienceLogic, the Reston, Virginia-based AIOps vendor that raised $105 million a year ago. “As large enterprises shift workloads to the cloud while managing on-prem resources, new tools are paramount to deliver service visibility and faster incident resolutions made better by advanced AI/ML technologies,” stated Dave Link, ScienceLogic’s founder and CEO.

There are many more companies, large and small, hoping to get a piece of the growing market for observability and AIOps tools. Keep Datanami bookmarked to find out what vendors are doing, and how customers are benefiting from these new capabilities.

Related Items:

Companies Drowning in Observability Data, Dynatrace Says

Cribl Seeks Control of Observability Data Run Amok

Who’s Winning In the $17B AIOps and Observability Market