Momentum is building around Velox, a new C++ acceleration library that can deliver a 2x to 8x speedup for computational engines like Presto, Spark, and PyTorch, and likely others in the future. The open source technology was originally developed by Meta, which today submitted a paper on Velox to the International Conference on Very Large Data Bases (VLDB) taking place in Australia.
Meta developed Velox to standardize the computational engines that underly some of its data management systems. Instead of developing new engines for each new transaction processing, OLAP, stream processing, or machine learning endeavor–which require extensive resources to maintain, evolve, and optimize–Velox can cut through that complexity by providing a single system, which simplifies maintenance and provides a more consistent experience to data uses, Meta says.
“Velox provides reusable, extensible, high-performance, and dialect-agnostic data processing components for building execution engines, and enhancing data management systems,” Facebook engineer Pedro Pedreira, the principal behind Velox, wrote in the introduction for the Velox paper submitted today at the VLDB conference. “The library heavily relies on vectorization and adaptivity, and is designed from the ground up to support efficient computation over complex data types due to their ubiquity in modern workloads.”
Based on its own success with Velox, Meta brought other companies, including Ahana, Voltron Data, and ByteDance, to assist with the software’s development. Intel is also involved, as Velox is designed to run on X86 systems.
The hope is that, as more data companies and professionals learn about Velox and join the community, that Velox will eventually become a regular component in the big data stack, says Ahana CEO Stephen Mih.
“Velox is a major way to improve your efficiency and your performance,” Mih says. “There will be more compute engines that start using it….We’re looking to draw more database developers to this product. The more we can improve this, the more it lifts the whole industry.”
Mih shared some TPC-H benchmark figures that show the type of performance boost users can expect from Velox. When Velox replaced a Java library for specific queries, the wall clock time was reduced anywhere from 2x to 8x, while the CPU time dropped between 2x and 6x.
They key advantage that Velox brings is vectorized code execution, which is the ability to process more pieces of code in parallel. Java does not support vectorization, whereas C++ does, which makes many Java-based products potential candidates for Velox.
Mih compared Velox to what Databricks has done with Photon, which is a C++ optimization layer developed to speed Spark SQL processing. However, unlike Photon, Velox is open source, which he says will boost adoption.
“Usually, you don’t get this type of technology in open source, and it’s never been reusable,” Mih tells Datanami. “So this can be composed behind database management systems that have to rebuild this all the time.”
Over time, Velox could be adapted to run with more data computation engines, which will not only improve performance and usability, but lower maintenance costs, writes Pedreira and two other Facebook engineers, Masha Basmanova and Orri Erling, in a blog post today.
“Velox unifies the common data-intensive components of data computation engines while still being extensible and adaptable to different computation engines,” the authors write. “It democratizes optimizations that were previously implemented only in individual engines, providing a framework in which consistent semantics can be implemented. This reduces work duplication, promotes reusability, and improves overall efficiency and consistency.”
Velox uses Apache Arrow, the in-memory columnar data format designed to enhance and speed up the sharing of data among different execution engines. Wes McKinney, the CTO and co-founder of Voltron Data and the creator of Apache Arrow, is also committed to working with Meta and the Velox and Arrow communities.
“Velox is a C++ vectorized database acceleration library providing optimized columnar processing, decoupling SQL or data frame front end, query optimizer, or storage backend,” McKinney wrote in a blog post today. “Velox has been designed to integrate with Arrow-based systems. “Through our collaboration, we intend to improve interoperability while refining the overall developer experience and usability, particularly support for Python development.”
These are still early days for Velox, and it’s likely that more vendors and professionals will join the group. Governance and transparency are important aspects to any open source project, according to Mih. While Velox is licensed with an Apache 2.0 license, it has not yet selected an open source foundation to oversee its work, Mih says.
Related Items:
Ahana Launches ‘Forever Free’ Presto Service, Series A Top-Off
Databricks Scores ACM SIGMOD Awards for Spark and Photon
Voltron Data Takes Flight to Unify Arrow Community
Editor’s note: This article has been corrected. Wes McKinney is the CTO and co-founder of Voltron Data, not the CEO. Datanami regrets the error.
March 29, 2024
- Domo Expands Native Integration Capabilities within Snowflake Data Cloud
- OpenPipe Secures $6.7M in Seed Funding to Revolutionize Developer Access to Custom LLMs
March 28, 2024
- Elastic Announces 2023 Elastic Excellence Awards Winners
- Woolpert Acquires Ireland-Based Murphy Geospatial, a Leading European Geospatial Solutions Firm
- WiDS Livermore Conference Attendees Network, Share Research and Absorb Wisdom
- Observe Announces $115M In Series B Financing
- Lightning AI’s New Thunder Compiler Boosts AI Development Efficiency by 40%
- Intel Gaudi 2 Remains Only Benchmarked Alternative to NV H100 for GenAI Performance
- Appen Launches Solution for Enterprises to Customize LLMs
- MineOS Unveils AI Asset Discovery
- Cloudera Survey Reveals 90% of IT Leaders Believe that Unifying the Data Lifecycle on a Single Platform is Critical for Analytics and AI
- Snowflake Enhances Secure, Cross-Cloud Collaboration for High Value Business Outcomes with Snowflake Data Clean Rooms
- Domo Announces Winners of the 2024 Community Ovation Awards
March 27, 2024
- New MLPerf Inference Benchmark Results Highlight the Rapid Growth of Generative AI Models
- Qlik Advances Real-time Data Analytics with Solace PubSub+ Platform Integration
- Samsung Unveils Expanded CXL Memory Module Portfolio at Memcon 2024, Enhancing AI and HPC
- Celestial AI Closes $175M Series C Funding Round Led by US Innovative Technology Fund
- Databricks Launches DBRX: A New Standard for Efficient Open Source Models
- Astronomer Unveils New Capabilities in Astro to Streamline Enterprise Data Orchestration
- DataVisor Introduces Enhanced Anti-Money Laundering Solution to Support Financial Institutions
Most Read Features
Sorry. No data so far.
Most Read News In Brief
Sorry. No data so far.
Most Read This Just In
Sorry. No data so far.
Sponsored Partner Content
-
Supercharge Your Data Lake with Spark 3.3
-
Learn How to Build a Custom Chatbot Using a RAG Workflow in Minutes [Hands-on Demo]
-
Overcome ETL Bottlenecks with Metadata-driven Integration for the AI Era [Free Guide]
-
Gartner® Hype Cycle™ for Analytics and Business Intelligence 2023
-
The Art of Mastering Data Quality for AI and Analytics
Sponsored Whitepapers
Contributors
Featured Events
-
Data Universe
April 10 - April 11New York United States -
Call & Contact Center Expo
April 24 - April 25Las Vegas NV United States -
AI & Big Data Expo North America 2024
June 5 - June 6Santa Clara CA United States -
AI Hardware & Edge AI Summit 2024
September 10 - September 12San Jose CA United States