The Linux Foundation Announces General Availability of Delta Lake 3.0

SAN FRANCISCO, Oct. 19, 2023 — The Linux Foundation is excited to announce the general availability of Delta Lake 3.0, the groundbreaking release of the renowned open-source Delta Lake project.

Developed under the banner of The Linux Foundation, this version of Delta Lake unifies lakehouse storage formats and gives users flexibility and control over their data.

Key Features of Delta Lake 3.0:

1. Delta Universal Format (UniForm): Companies have long grappled with the decision of which storage format to standardize in their pursuit of creating a seamless data lakehouse. UniForm is an innovative feature that facilitates reading data in the format demanded by the application, ensuring heightened compatibility and a broader ecosystem. With UniForm, the challenges of choosing between Apache Iceberg or Apache Hudi are erased. Delta now effortlessly generates the essential metadata for both, eliminating the need for manual conversions, providing an unmatched layer of flexibility and efficiency.
2. Delta Kernel: The expansive connector landscape for Delta often poses challenges for developers. Focused on streamlining the process of building Delta connectors, Delta Kernel offers a simplified API, ensuring that developers can harness the full suite of Delta’s offerings without being entangled in its underlying complexities. By masking the intricate details of the Delta protocol specification, developers can concentrate on leveraging the potential of Delta.
3. Dramatically Faster MERGE and Updates: With Delta 3.0, MERGE now better leverages data skipping, the ability to use the insert-only code path in more cases, and an overall improved execution to achieve up to 2x better performance in various scenarios. DELETE operations are now more than 2x faster. Faster DELETE operations translate to reduced wait times when purging or updating data, enabling more agile data management and processing.

“The Linux Foundation has always championed open source innovation and collaboration,” said Jim Zemlin, Executive Director of the Linux Foundation. “With the release of Delta Lake 3.0, we see a testament to the power of community-driven development in the data world. This milestone represents not just technological advancement with industry-changing features like UniForm and Delta Kernel, but also the potential of open ecosystems to drive change and foster innovation.”

Users worldwide can now leverage Delta Lake 3.0 to streamline their data management processes, optimize performance, and enhance flexibility. Eager to delve deeper into the feature set and discover more about Delta Lake 3.0? Explore the detailed release notes here. For more information on Delta Lake 3.0 and its revolutionary features, read this blog.

About The Linux Foundation

Founded in 2000, the Linux Foundation and its projects are supported by more than 2,950 members. The Linux Foundation is the world’s leading home for collaboration on open source software, hardware, standards, and data. Linux Foundation projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, ONAP, Hyperledger, RISC-V, and more. The Linux Foundation’s methodology focuses on leveraging best practices and addressing the needs of contributors, users, and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org.

Source: The Linux Foundation