Zilliz Announces Key Contributions to Milvus 2.1

SAN FRANCISCO, Aug. 4, 2022 — Zilliz, whose founders created the Milvus open-source project, today announced major contributions to the Milvus 2.1 release. The added functionality further bridges the gap between data pools, removes data silos, and offers performance and availability enhancements to address developers’ most common concerns. Milvus is one of the world’s most advanced vector databases, capable of managing massive quantities of both structured and unstructured data to accelerate the development of next-generation data fabric.

“When it comes to unstructured data, solutions offered by industry incumbents tend to be add-on capabilities or tools in a legacy database management system, whereas Milvus is designed around unstructured data from day one and is now offering more built-in capabilities to unlock more powerful and integrated data processing.”

A graduated-stage open-source project under the LF AI & Data Foundation, Milvus is built for scalable similarity search and used by a wide array of enterprises across industries. It embraces distributed architecture and can easily scale as data volumes and workloads increase. Highly scalable, reliable, and exceptionally fast, Milvus supports DML operations (adding, deleting, updating) and near real-time search of vectors on a trillion-byte scale.

With this 2.1 update, Milvus sees significant improvement in its performance, reducing search latency on million-scale datasets to five milliseconds, while further simplifying deployment as well as ops workflow.

Bridging Gaps and Improving Performance

Machine learning is producing vast pools of scalar and vector data on a daily basis. With the introduction of more scalar data types, Milvus 2.1 is bridging this critical gap between data pools.

“Data silos can now be better integrated and linked, enabling businesses to unlock the full potential of their data,” said Milvus project maintainer Xiaofan James Luan, who also serves as the director of engineering at Zilliz. “When it comes to unstructured data, solutions offered by industry incumbents tend to be add-on capabilities or tools in a legacy database management system, whereas Milvus is designed around unstructured data from day one and is now offering more built-in capabilities to unlock more powerful and integrated data processing.”

Zilliz’s contributions to the 2.1 release specifically include:

An overall performance boost including reduced latency; highly improved throughput for small-NQ application scenarios, such as reverse image search and intelligent chatbot; support of multiple memory replicas for small tables to increase throughput; and 2x increase in search performance.

Improved scalar data processing that adds Varchar into supported data types and supports creating indexes for scalar data, taking hybrid search to a more intuitive level.

Production-grade enhancements and higher availability, with clearer monitoring metrics for observability, easier and more diverse deployment options including embedded Milvus for simple deployment and Ansible for offline deployment, integration that supports Kafka as log storage, and enhanced security supporting password protection and TLS connection.

A developer-friendly ecosystem in the making that includes more tutorials for building real-world applications, connecting Milvus with open-source vector data ETL framework Towhee; and that adds Feder, an open-source tool that helps Milvus users select the index best suited to their application scenario by visualizing the process of vector similarity search.

In addition to the integration and security features enumerated, Milvus will provide more functionalities essential to modern security mechanisms, including ACL (Access Control Lists) and advanced encryption methods.

Commitment to Open-Source Ecosystems

“As data infrastructure for unstructured data, Milvus is revolutionary because it processes vector embeddings and not just strings. In the future, Zilliz, the company founded by the creators of Milvus, seeks to build an ecosystem of solutions around Milvus, and some of the projects that will contribute to this have already surfaced, including Towhee, our open-source vector data ETL framework, and Feder, an interactive visualization tool for unstructured data. With Milvus 2.1 and the new demos, users can see how these products can come together to solve a series of problems that involve unstructured data,” added Luan.

Zilliz is committed to the developer community and will continue to contribute to open-source projects like Milvus. The company’s technology has broad applications spanning new drug discovery, computer vision, recommender engines, chatbots, and much more.

About Zilliz

Zilliz is a leading vector database company for production-ready AI. Built by the engineers who created Milvus, the world’s most popular open-source vector database, Zilliz is on a mission to unleash data insights with AI. The company builds next-generation database technologies to help organizations rapidly create AI/ML applications and unlock the potential of unstructured data. By taking the burden of complex data infrastructure management off of its users, Zilliz is committed to bringing the power of AI to every corporation, every organization, and every individual.

Source: Zilliz