Comet Partners with Snowflake to Bring Governed Reproducibility of Datasets for Machine Learning with Snowpark

NEW YORK, June 16, 2023 — Comet has announced a partnership with Snowflake to deliver groundbreaking solutions that will empower data scientists to build superior models with greater speed, bolstering data-driven decision-making.

Reproducibility of ML models requires versioning of the code, hyperparameters, and data. Keeping data within Snowflake, instead of moving it to a different system, is imperative for eliminating data silos and complexity. Comet’s partnership with Snowflake will integrate its solutions with Snowflake’s single, unified platform so developers can track and version their Snowflake queries and datasets within their own Snowflake environment. This integration facilitates easy lineage tracing of models and performance, providing better visibility and understanding of the development process and the impact of data changes on model performance.

“We look forward to partnering with Comet to provide machine learning teams with solutions that address operations challenges that arise when developing and deploying models,” said Torsten Grabs, Senior Director, Product Management, Snowflake. “By combining the power of Comet’s ML platform with the Snowflake Data Cloud, joint customers can now reliably track all the datasets used across their different ML workflows, allowing them to build ML models in a repeatable, auditable and secure manner.”

The partnership between Snowflake and Comet marks a significant advancement in supporting ML workflows, integrating a top MLOps tool offering and enhanced support for customers who rely on Snowflake to securely store, manage, and process their data. By leveraging Snowflake data for model building, customers can now benefit from a more streamlined and transparent model development process.

“We’re excited to partner with Snowflake to provide developers building ML applications with a comprehensive overview of their model development process,” said Gideon Mendels, CEO of Comet. “With the integration of Comet and Snowflake, we’re enabling customers to better track and version their Snowflake queries and datasets, ultimately improving their understanding of how their models were developed and how changes made to the data affect model performance.”

The powerful integration comes on the heels of Comet’s recent release of a suite of tools and integrations designed to accelerate Large Language Model workflow for Data Scientists, further underscoring Comet’s commitment to provide teams with better ways to leverage ML.

Integration Benefits

The integration of Comet’s data science platform with the Snowflake Data Cloud enables data scientists to use Snowflake as their primary data source while tracking and managing their experiments using Comet. This collaboration allows data scientists to version and reproduce their ML workflows, ensuring model quality and compliance are maintained in production.

Comet’s Python SDK has been updated to allow the logging of Snowpark DataFrames as Comet Artifacts, which saves the query, metadata, and a snapshot of the DataFrame as a Comet Artifact that can be viewed in the Comet UI. Furthermore, Artifacts are linked to Comet Experiments that contain all the metadata relevant to the model, which enables developers to view their data together with their model metrics.

The Comet-Snowflake integration offers additional benefits, including effortless versioning and artifact tracking within Snowflake, enhanced MLOps capabilities for managing large-scale projects, and a simplified workflow for data scientists.

For more information on Comet’s integration, please visit the Comet blog.

About Comet

Comet provides the leading platform for managing Machine Learning models all the way from early experimentation to monitoring them in production. It provides tools for experiment tracking, artifact management, model registries, collaboration, visualization and supports integration with all the popular ML frameworks. Comet’s mission is to help Data Scientists and ML teams accelerate the development and deployment of ML models and help companies achieve business value from AI.

Source: Comet