Matillion + Snowflake: Increase Your Speed to Insight with Data Modernization

Shutterstock 233848510

Business analytics tools have made it possible for companies of all sizes to gain tremendous insights from the data they collect. Whether it’s to improve decision making,  optimize operations, improve marketing campaigns, or automate rote tasks, analysis of company data and transforming it to meaningful information is an important key to success in today’s business environment.

Of course, the information is best when it’s collected, analyzed and leveraged in a timely manner. Yet, many businesses struggle with the time it takes to ingest, integrate, and clean up data from different sources so that it can be used to drive business outcomes.

While many companies use cloud data warehouses such as Snowflake and business analytics tools such as Tableau and PowerBI, they are missing a very important piece of the data extraction puzzle. Similar to BI and storage, cloud-native ELT is  more accessible for fast growing, modern businesses than it ever has been before.

Working Smart: Preparing and Transforming Data from Multiple Sources

When it comes to data modernization, a big challenge is designing the queries and workflows that best leverage the available data. The tedious part is preparing, transforming, and integrating the data from different systems so that it can be analyzed together.

The fact that different databases have different parameters, such as varying date and time formats, the length of data strings, and field ordering in tables, adds to complexity.

Organizations primarily use a process called ETL, or Extract-Transform-Load to accomplish these tasks. It’s the process by which data is extracted from its original source – database, applications, file systems, and more . – transformed to a standard format, and loaded into a data warehouse. Prior to the advent of automated tools, ETL used to require a tremendous amount of hand-coding to build data pipelines and transform data. In fact, without  cloud ETL, data preparation can occupy close to half of the working time of data scientists.

Nearly half (45%) of time spent on data analytics projects is dedicated to data preparation, instead of on more strategic, high-value tasks

Fortunately, integrating, cleaning and standardizing data is an ideal task to automate and a number of products have emerged to provide automated ETL. Like many new technologies, the first iterations of ETL tools were complex and expensive. They required manual programming and collection of the machine learning data. For this reason, ETL was once only accessible to large organizations due to cumbersome integration and high cost.

Today, as the technology has matured, cloud native ETL applications such as Matillion are accessible to all sizes of organizations. ETL today does not require a large investment. Automated data cleansing software today leverages the power of cloud processing engines by transforming data within the warehouse itself in a pushdown ETL approach, allowing implementation of  pricing models based on actual usage of the software. All of this means that cloud ETL tools are accessible to fast growing, modern businesses of any size.

These low code/no-code solutions allow any organization to automate data integration and transformation tasks, for example, by creating customized data workflows.  Furthermore, the visual low-code/no-code interface creates a very short learning cycle for new users coming into the business. Data scientists can quickly reduce the time they spend cleaning data—or maintaining the systems that do the same—and increase the time spent supporting the real work of business analytics, reporting and decision making.

Data Modernization:  No Code, No Gaps, No Fuss

The Matillion cloud-native ETL system is a low-code/no-code platform that allows users to instantly see the fields in their data tables and work with a drag-and-drop interface to create data pipelines and data transformations that can’t be automatically configured by software. Furthermore, by leveraging the power and scale of the cloud, Matillion is able to integrate and transform data much faster than most on-premises data systems. Cloud processing simply provides faster throughput for any given task with the elasticity to expand and reduce processing capacity as needed.  The system includes a large set of out-of-the-box connectors to leading applications, as well as a connector creation toolkit that offers the full power of programmable systems, but without the steep learning curve or complexity of workflow creation. The toolkit allows companies to rapidly create custom connectors using a low-code/no-code model that is easy to learn.

Using a low-code/no-code connector toolkit means that companies can easily import and integrate data, even from bespoke sources. Legacy databases that might be difficult to parse, or are “too small” to bother with, are easy to connect to and their data is easy to transform using Matillion. Companies can now get access to data that might have been ignored if they had to manually handle the process.

Case Study

About: TUI is a major European travel conglomerate that brings together multiple brands and businesses.

Challenge: Brands, subsidiaries, and local branches of the company all have their own computer systems, including multiple languages and formats. For example, every travel agency has its own email system, some of which are proprietary or niche applications used by just one small group within the conglomerate.

Impact: Without automation, TUI experienced gaps in their data and analysis, because it simply wasn’t  worthwhile to cleanse every niche app database.

Solution: Matillion enabled TUI to create custom connectors, often just in a few minutes, with the connector API toolkit. Rather than asking their subsidiaries to move to another tool, TUI  was simply and easily able to achieve cloud data integration even from niche email marketing platforms from subsidiaries into their Snowflake data warehouse. They then used the imported data to create a complete picture of bounce rates, subscribers, and other funnel activity.

Matillion for Snowflake: Workflow and ETL integration speed and security

Fast integration is important for organizations of any size. Because Matillion has a modular, low-code/no-code design, it is easy to learn, and companies do not need to hire dedicated developers to manage the system. Matillion is a browser-based interface with a rapid onboarding and training process for data analytics teams of any size.

From an IT perspective, Matillion easily integrates with most major cloud data warehouses and data lakes, including Snowflake. And, with years of experience in cloud-native ETL, Matillion has also developed a large library of connectors to most databases and other data sources and applications, both on premises and in the cloud, including Oracle, SAP, Salesforce, and Marketo. The Matillion platform uses an ANSI compliant SQL configuration that is familiar to most IT professionals.

Matillion is deployed in a company’s own public cloud environment (AWS, Azure or Google Cloud), but does not store any data or information about the  data or data structures on the deployed Matillion instance. All data, workflows, data structure information, and metadata is stored in the company’s own virtual private cloud. In other words, all of the security, compliance and data governance requirements within the existing IT system remain fully intact with no added attack surface area.

Matillion + Snowflake: Let Data Scientists Actually Do Data Science

It seems obvious that data scientists want to spend their time modeling and analyzing data. But if you don’t have a data management strategy and  an ETL tool in place, the data scientists often spend as much as half of their time just on preparing and transforming data for analytics (Source: IDG Research MarketPulse survey). That’s a lot of underutilized resources, not to mention dissatisfied data scientists who would rather be doing actual data analysis and creating actionable insights for the company.

Case Study

About: Uberflip is a leading marketing content experience platform. Uberflip brings in data from multiple sources including marketing data from systems such as Marketo and Salesforce, customer service data from Zendesk and Bombara, and product information from Jira, etc. and an ERP system. It integrates the data in Snowflake and displays reports and analytics with Looker.

Challenge: Integrating information from new datasets was taking five weeks of mostly tedious manual work from the data scientists, wasting valuable time and slowing down the integration and analysis.

Solution: With Matillion’s low-code/no-code workflow builder, it now takes Uberflip just one day to integrate new datasets.

Speeding up business intelligence with modern, cloud-native ETL

Using manual processes for data transformation is a drag on the business, both in terms of the hours it’s draining for the BI team and in terms of response times to changes.

Using Matillion and Snowflake together can speed up the process of integrating all of your data to glean actionable business insights. Matillion can be up and running for your business in a matter of days. Schedule a demo to see Snowflake and Matillion in action.

Get a demo