Speedb’s Data Storage Engine Goes Open Source

Speedb has announced it is going open source with its enterprise data storage engine billed as a drop-in replacement for RocksDB.

The company says making this technology available through open source brings an opportunity for the development community to take part in shaping the data architecture of the future. Key features from the enterprise version of Speedb are being made available to the open source community to support the needs of today’s data-intensive workloads, the company stated.

“We’re 100% committed to building a robust developer community around a modern data engine for modern applications, where data volumes often grow large and unwieldy,” said Adi Gelvan, co-founder and CEO of Speedb, in an email to Datanami. “We need developers to help us build the best storage engine we can. We want to ask the community what it wants, receive their pull requests, and refine options we may not even have considered. With the feedback we get from the community we can build something really special.”

The Speedb data engine is a key-value store, which is a NoSQL database composed of a key and an associated data value. Used in cases where horizontal scaling is required, key-value stores are especially hot in metadata management where a flood of metadata is impacting the performance of databases, streaming apps, and analytics.

RocksDB, an open source project since 2012, is the de facto standard embedded key-value store, according to Speedb. However, while working with RocksDB at storage vendor Infinidat, the founders of Speedb found it begins to stall when metadata storage approaches 50GB. Other RocksDB issues include code instability and write amplification, along with the need for sharding and performance tuning, says Speedb.

Speedb redesigned key parts of RocksDB including a compaction algorithm that reduces the Write Amplification Factor (WAF) from ~30 to ~5 – as well as redesigning its I/O and flow mechanisms. RocksDB users can embed the Speedb data engine by changing a few lines of code to boost metadata memory performance and reduce operational tasks, according to Speedb.

“We’ve been using Speedb internally for a while now, and we’re very excited for the promise it shows over RocksDB, in memory management, compaction, and configurability,” said Haikal Pribadi, CEO at Vaticle, creators of the strongly-typed database, TypeDB, and its query language, TypeQL. “Most importantly, having the dedicated support of Speedb in our development efforts on top of the storage infrastructure has been invaluable. Speedb already has made a huge difference to our team.”

A still from an introductory video shows the capabilities of Speedb. Source: Speedb

Speedb will continue selling its enterprise edition through Redis and other OEMs for those developers with larger and more complex applications. The company’s enterprise edition offers additional features, higher scalability, support services, and tuning. The company claims users of its enterprise edition can generate up to double throughput with half the latency and double the capacity compared to the same deployment with RocksDB.

By offering its data engine through open source, the company hopes to improve its product through collaboration with the developers who know it best. The Speedb open source community will serve as a hub for Speedb and RocksDB users to collaborate while sharing knowledge, issues, and solutions, the company says.

“By going open source, Speedb can help far more developers to remove the key barriers to data performance and take their hyperscale data operations to the next level, backed by a supportive developer community,” said CEO Gelvan. “We’ll be making ongoing additions of our enterprise functionality to the open-source edition and are looking forward to embracing the community assistance and collaborating with the many creative minds in the developer realm that are also searching for more efficient ways to rev their data engines.”

Speedb’s open source edition is available at GitHub and Speedb’s dev site.

Related Items:

Speedb Picks Up Where RocksDB Leaves Off

Second Half 2022 Tech Predictions for Data and AI

Speedb Secures $4M and Launches on Redis to Reveal New Data Engine Technology