

(Jezper/Shutterstock)
If the exhibitors at last week’s Strata + Hadoop World expo are any indication of what’s happening down on the street, data cataloging is evolving from a nice-to-have into a necessity for organizations looking to capitalize on big data.
Hadoop’s so-called “junk drawer” problem has been well-documented. It stems largely from the flexible schema-on-read approach, where data is structured only when it’s finally accessed from the data lake, as opposed to the traditional ETL approach of transforming data when it’s originally loaded into the data warehouse.
In short, getting data into Hadoop is easy, but finding it and getting it back out again can be hard. All sorts of vendors are now looking to address this dilemma, which touches many aspects of big data analytics, including data quality and security. Having a catalog of the data stored in Hadoop seems like a good idea, and there are a number of vendors providing that.
Alex Gorelik, CEO and founder of Waterline Data, which provides data cataloging software for Hadoop and other big data systems, says data professionals are reluctant to open Hadoop to downstream users without a better accounting of the actual data.
“The data lake looks like a flea market,” Gorelik tells Datanami. “It’s all in there somewhere, but how do you find it? It’s a problem for data scientists and data stewards because they can’t give people access until they know what’s in there.”
Gorelik says that while open source tools like Apache Atlas, which is backed by Hortonworks (NASDAQ: HDP), and Cloudera Navigator provide a good technical foundation for addressing data cataloging and master data management (MDM) challenges, they don’t go far enough to solve the problem. Waterline addresses the problem by using “tags” to track the lineage of every piece of data.

Data lakes are resembling “flea markets” says Alex Gorelik, CEO and founder of Waterline Data (Toniflap/Shutterstock)
With Waterline, Hadoop users can continue ingesting data as they did before, while relying on the software to keep it somewhat organized. Apache Lucene sits under the covers to power searches, while an Amazon-like user interface and “shopping cart” process lets analysts check-out when they’ve found their data.
It’s not a license to be messy with your data, but at least it takes the burden off of users to manually track their data. “People used to have careful directories. But these days, they can’t keep track of their directories,” Gorelik says. “You have millions of files. You should organize them as well as you can. [With Waterline software] it doesn’t matter where the file is, as long as you can find it.”
Collibra is another master data management (MDM) software vendor helping customers keep track of their Hadoop-resident data using the catalog approach. The company, which recently moved its headquarters to New York City, has an eight-year history of providing data governance solution to customers in healthcare, financial services, and other industries.
“What we have is a technology platform that has the capability to keep track of processes around data, the metadata and organization and roles and responsibility for data,” Daniel Sholler, director of product marketing at Collibra, tells Datanami. “We keep track of all the technical connections of all the data because you need to know that stuff. But it turns out that stuff isn’t the interesting stuff.”
Instead, Collibra exposes a set of applications that make it relatively easy for end users to get access to data, if they are authorized to access it. That’s the “interesting” stuff that Sholler was referring to. Data access is one component of a collection of data governance solutions that Collibra is offering, and the scope of that offering will expand in the coming weeks.
Another vendor that’s plying the fruitful waters of data cataloging is Alation. The company originally designed its product to “learn” about data connections by observing how analysts interact. But just providing data cataloging wasn’t enough, the company says. So last week Alation announced that in its version 4.0 update, it will also track queries that run along with the data that’s collected.
Tracking queries and data, says Alation CTO Venky Ganti, will provide critical context that’s required for addressing the needs of data stewards and customers, including answering questions like “Where can I find data to answer my question?” “Can I trust this data?” “What are the data semantics in order to use it?” and “Who can answer my question about this data set.”

Will you organize the data in your lake by hand, or rely on power tools to simplify the task? (cherezoff/Shutterstock)
“Experts who understand certain datasets often play the stewardship role of ensuring that data consumers can make accurate and effective use of data,” Ganti says in a blog post. “More recently, data governance initiatives have started to assign formal stewardship responsibility.”
Other companies offering data cataloging functionality include Podium Data, which announced a $9.5-million Series A round just prior to the show. Zaloni also unveiled its Bedrock Data Lake Manager (DLM) product, which uses data cataloging to help manage storage more effectively. At Strata, it launched a new version of Mica, its data preparation tool, which introduces a new “shopping cart”-like experience.
That “shopping cart” metaphor was heard often on the Strata expo floor during discussions of data catalogs and big data management. You can expect to see that show up in MDM and data quality tools more often.
Informatica, the big dog of last-gen ETL tools that’s hungering for a piece of the big data pie, also updated its data lake management product, called Data Lake Management, to include more capabilities. Specifically, the product combines data cataloging, stream data capture, Hadoop job management, security, and cloud connectors in a single unified product.
The lack of a centralized data lake management point eats up analysts’ time and hurts productivity, says Amit Walia, executive vice president and chief product officer for Informatica. “Ease of use and a delightful user experience along with robust governance and metadata capabilities are critical for getting business value out of data lakes,” he says in a statement.”
According to Gartner analysts Guido De Simoni and Roxane Edjlali, enterprise metadata management, including data cataloging, has become a “required discipline.” “Failure to recognize this will lead to sustained siloed behavior and loss of business value,” they wrote earlier this year.
While data silos will inevitably be with us for a while, we don’t have to behave as if the data is trapped in a single location. As the Gartner analysts rightly point out, organizations that can get a unified view of their data will find greater business value. It’s becoming clear that data catalogs will be one way of providing that visibility.
Related Items:
Delivering on the Data Lake Promise
8 Tips for Achieving ROI with Your Data Lake
Taming the Wild Side of Hadoop Data
July 1, 2025
- Nexdata Presents Real-World Scalable AI Training Data Solutions at CVPR 2025
- IBM and DBmaestro Expand Partnership to Deliver Enterprise-Grade Database DevOps and Observability
- John Snow Labs Debuts Martlet.ai to Advance Compliance and Efficiency in HCC Coding
- HighByte Releases Industrial MCP Server for Agentic AI
- Qlik Releases Trust Score for AI in Qlik Talend Cloud
- Dresner Advisory Publishes 2025 Wisdom of Crowds Enterprise Performance Management Market Study
- Precisely Accelerates Location-Aware AI with Model Context Protocol
- MongoDB Announces Commitment to Achieve FedRAMP High and Impact Level 5 Authorizations
June 30, 2025
- Campfire Raises $35 Million Series A Led by Accel to Build the Next-Generation AI-Driven ERP
- Intel Xeon 6 Slashes Power Consumption for Nokia Core Network Customers
- Equal Opportunity Ventures Leads Investment in Manta AI to Redefine the Future of Data Science
- Tracer Protect for ChatGPT to Combat Rising Enterprise Brand Threats from AI Chatbots
June 27, 2025
- EarthDaily Ignites a New Era in Earth Observation with Landmark Satellite Launch
- Domo Deepens Collaboration with Snowflake to Accelerate AI-Driven Analytics and Data Integration on the AI Data Cloud
- AIwire Launches Annual People to Watch Program
June 26, 2025
- Thomson Reuters: Firms with AI Strategies Twice as Likely to See AI-driven Revenue Growth
- DataBahn Raises $17M Series A to Advance AI-Native Data Pipeline Platform
- BCG Report: Companies Must Go Beyond AI Adoption to Realize Its Full Potential
- H2O.ai Breaks New World Record for Most Accurate Agentic AI for Generalized Assistants
- Foresight Raises $5.5M Seed Round to Bring Unified Data and AI to the Private Market
- Inside the Chargeback System That Made Harvard’s Storage Sustainable
- What Are Reasoning Models and Why You Should Care
- Databricks Takes Top Spot in Gartner DSML Platform Report
- Snowflake Widens Analytics and AI Reach at Summit 25
- Why Snowflake Bought Crunchy Data
- LinkedIn Introduces Northguard, Its Replacement for Kafka
- Change to Apache Iceberg Could Streamline Queries, Open Data
- Agentic AI Orchestration Layer Should be Independent, Dataiku CEO Says
- Top-Down or Bottom-Up Data Model Design: Which is Best?
- The Evolution of Time-Series Models: AI Leading a New Forecasting Era
- More Features…
- Mathematica Helps Crack Zodiac Killer’s Code
- AI Agents To Drive Scientific Discovery Within a Year, Altman Predicts
- ‘The Relational Model Always Wins,’ RelationalAI CEO Says
- Confluent Says ‘Au Revoir’ to Zookeeper with Launch of Confluent Platform 8.0
- Solidigm Celebrates World’s Largest SSD with ‘122 Day’
- DuckLake Makes a Splash in the Lakehouse Stack – But Can It Break Through?
- The Top Five Data Labeling Firms According to Everest Group
- Supabase’s $200M Raise Signals Big Ambitions
- Data Prep Still Dominates Data Scientists’ Time, Survey Finds
- Toloka Expands Data Labeling Service
- More News In Brief…
- Astronomer Unveils New Capabilities in Astro to Streamline Enterprise Data Orchestration
- Databricks Unveils Databricks One: A New Way to Bring AI to Every Corner of the Business
- BigID Reports Majority of Enterprises Lack AI Risk Visibility in 2025
- Snowflake Openflow Unlocks Full Data Interoperability, Accelerating Data Movement for AI Innovation
- Astronomer Introduces Astro Observe to Provide Unified Full-Stack Data Orchestration and Observability
- Seagate Unveils IronWolf Pro 24TB Hard Drive for SMBs and Enterprises
- Gartner Predicts 40% of Generative AI Solutions Will Be Multimodal By 2027
- Databricks Donates Declarative Pipelines to Apache Spark Open Source Project
- Code.org, in Partnership with Amazon, Launches New AI Curriculum for Grades 8-12
- BigBear.ai And Palantir Announce Strategic Partnership
- More This Just In…