Matillion ETL Supports Data Lineage Tools with Metadata API Integrations

Shutterstock 328608269

At Matillion, we spend a lot of our time talking to users. We’re especially interested in hearing your thoughts and opinions on Matillion ETL; what you love about it, and how we could make it better.

In these conversations, a topic that comes up time and time again is data governance. In particular, how Matillion handles data lineage.

You’ve told us that you want Matillion ETL to be a platform for your chosen lineage tool (including in-house lineage reporting). We believe that Matillion ETL should work seamlessly with best of breed partner applications, and data lineage is no different.

Our best of breed lineage integrations

We listened to your feedback. That’s what we developed and released Matillion’s Metadata API in spring of 2021. Now, we’re pleased to announce support for deep integrations with the following data cataloguing and governance partners on that API, including:

• Collibra, 
• Zeenea
• and OvalEdge 

And this is only the start. We’ve got many more integrations planned, allowing you to pair Matillion ETL with the data lineage tools of your choice.

Start here to find out more about our Collibra integration.

Matillion ETL’s metadata API

In addition to our work with data management partners, we’ve also been busy enhancing data lineage functionality in our RESTful Metadata API. This API service allows Matillion ETL users to access the lineage log of each component in a Matillion ETL job in a programmatic and automated way.

For more details and examples, check out this blog post on `Use the Metadata API to Track SQL Executed Against Snowflake (Shared Job)` 

What’s next for data lineage in Matillion ETL?

From our user feedback, we know that Matillion ETL’s out of the box data lineage functionality doesn’t meet customers’ expectations. In fact, our own internal usage data tells us that it’s rarely used. With that in mind, we will remove the “Lineage” tab in Matillion ETL starting from version 1.63 onwards.

Why remove the “lineage” tab?

We’re really proud of Matillion ETL, and we don’t want to support a feature that isn’t loved or used by our customers.

We’re still committed to providing data lineage visibility in Matillion ETL, and we want Matillion ETL’s out of the box lineage functionality to be the best that it can be.
That’s why we’re launching a research project into the future of data lineage in Matillion ETL. We’d like lineage to come back into the product, bigger and better than before!

We need your help

If you’d like to be involved in shaping the next generation of out of the box lineage reporting in  Matillion, please get in touch with me via email: [email protected]. We’d love to hear your views and vision for the future of data lineage in Matillion ETL.