The concept of “observability” is well understood as it pertains to DevOps and site reliability engineering (SRE). But what does it mean in the context of data? According to Barr Moses, the CEO of data observability startup Monte Carlo, it’s all about being able to trust the data.
In the last decade or so, we’ve gotten really good at aggregating, storing, collecting, and analyzing data, Moses says. We’re able to move vast amounts of data from sources into data warehouses and data lakes, and utilize the data for dashboards and machine learning models.
“But whether we can actually trust the data is something we haven’t figured out yet,” Moses tells Datanami. “In companies, small to large, really the worst thing that can happen is when you start using the data, but it actually can’t be trusted.”
In fact, making decisions based on bad data can actively hurt a company. Considering all of the ways that humans and algorithms are consuming data in business today, using bad data is worse than doing nothing at all.
There are many ways that data can go bad. From data-entry or coding errors to malfunctioning sensors and data drift, the sources of data contamination are numerous. We can’t guarantee that data will never go bad, so the next best thing is to detect the bad data as soon as possible.
That’s the idea behind data observability. According to Moses, data observability borrows well-established observability concepts from DevOps and SRE in one key respect: the importance of monitoring the outputs of data systems to determine if something is going awry inside the system.
“If you think what DevOps or software engineer teams are tasked with, they have many applications and system infrastructure that they are tasked with making sure that they are up and running at all times,” Moses says. “It’s a very well-understood approach in DevOps, but it’s completely new to data. That’s what we call data observability.”
Monte Carlo’s data observability offering is structured around five pillars of observability:
- Freshness, or the timeliness of the data;
- Volume, of the completeness of the data;
- Distribution, which measures the consistency of data at the field level;
- Schema, relating to the structure of fields and tables;
- Lineage, or a change-log of the data.
“The only way to truly solve data observability is to do that in an end-to-end way,” Moses says. “So it includes the customers’ entire data stack. That includes cloud data lakes, data warehouse, ETL, BI, and machine learning models.”
The company, which has raised $41 million in venture capital to date, built connectors that pull data from each component of the stack, and monitors the data in a read-only manner. If the output begins to show signs of a problem – such as a dashboard reporting null values – the software will automatically generate an alert and send it via email, text, Slack or PagerDuty.
It’s about avoiding data downtime. Just as DevOps and SRE teams have instrumented their systems in an attempt to detect the faintest hint of a pending failure that could take production systems offline, data observability, as Monte Carlo practices it, takes a holistic approach to monitoring a range of data characteristics to figure out when good data is breaking bad.
Moses draws a parallel between what Monte Carlo is setting out to do and what New Relic, DataDog, and App Dynamics are doing in the DevOps and SRE space. While data infrastructure providers (i.e. vendors building databases, data lakes, data warehouses and ETL, BI, and ML tools) supply some monitoring capabilities for their products, it’s an industry best practice to tap third-party observability and monitoring tools to bring all of those components together into a comprehensive view.
“I think it’s important to have a new layer in the modern data stack,” says Moses, who was a management consultant for Bain & Company before founding Monte Carlo with Lior Gavish in 2019. “And part of the new layer is data observability. It’s important for it to be a third-party that can be integrated with all of these vendors and solutions…and also provide a third-party objective view” that can be trusted.
Today, the San Francisco company announced a partnership with cloud data warehouse provider Snowflake. The deal will see Monte Carlo become a “native” provider of data observability for Snowflake customers.
“Data is only powerful if you can actually trust it,” Moses says. “We firmly belief that’s a key part to building a strong data platform and data architecture. The Snowflake team has been really supportive of our vision and this new category of data observability. We’re exited to partner with them to bring this to the our customers.”
Related Items:
Do You Have Customer Data You Can Trust?
April 26, 2024
- Google Announces $75M AI Opportunity Fund and New Course to Skill One Million Americans
- Elastic Reports 8x Speed and 32x Efficiency Gains for Elasticsearch and Lucene Vector Database
- Gartner Identifies the Top Trends in Data and Analytics for 2024
- Satori and Collibra Accelerate AI Readiness Through Unified Data Management
- Argonne’s New AI Application Reduces Data Processing Time by 100x in X-ray Studies
April 25, 2024
- Salesforce Unveils Zero Copy Partner Network, Offering New Open Data Lake Access via Apache Iceberg
- Dataiku Enables Generative AI-Powered Chat Across the Enterprise
- IBM Transforms the Storage Ownership Experience with IBM Storage Assurance
- Cleanlab Launches New Solution to Detect AI Hallucinations in Language Models
- University of Maryland’s Smith School Launches New Center for AI in Business
- SAS Advances Public Health Research with New Analytics Tools on NIH Researcher Workbench
- NVIDIA to Acquire GPU Orchestration Software Provider Run:ai
April 24, 2024
- AtScale Introduces Developer Community Edition for Semantic Modeling
- Domopalooza 2024 Sets a High Bar for AI in Business Intelligence and Analytics
- BigID Highlights Crucial Security Measures for Generative AI in Latest Industry Report
- Moveworks Showcases the Power of Its Next-Gen Copilot at Moveworks.global 2024
- AtScale Announces Next-Gen Product Innovations to Foster Data-Driven Industry-Wide Collaboration
- New Snorkel Flow Release Empowers Enterprises to Harness Their Data for Custom AI Solutions
- Snowflake Launches Arctic: The Most Open, Enterprise-Grade Large Language Model
- Lenovo Advances Hybrid AI Innovation to Meet the Demands of the Most Compute Intensive Workloads
Most Read Features
Sorry. No data so far.
Most Read News In Brief
Sorry. No data so far.
Most Read This Just In
Sorry. No data so far.
Sponsored Partner Content
-
Get your Data AI Ready – Celebrate One Year of Deep Dish Data Virtual Series!
-
Supercharge Your Data Lake with Spark 3.3
-
Learn How to Build a Custom Chatbot Using a RAG Workflow in Minutes [Hands-on Demo]
-
Overcome ETL Bottlenecks with Metadata-driven Integration for the AI Era [Free Guide]
-
Gartner® Hype Cycle™ for Analytics and Business Intelligence 2023
-
The Art of Mastering Data Quality for AI and Analytics
Sponsored Whitepapers
Contributors
Featured Events
-
AI & Big Data Expo North America 2024
June 5 - June 6Santa Clara CA United States -
CDAO Canada Public Sector 2024
June 18 - June 19 -
AI Hardware & Edge AI Summit Europe
June 18 - June 19London United Kingdom -
AI Hardware & Edge AI Summit 2024
September 10 - September 12San Jose CA United States -
CDAO Government 2024
September 18 - September 19Washington DC United States