Zilliz Vector Database Research Featured at VLDB 2022
Zilliz has announced its latest research on vector databases was featured at the 48th International Conference on Very Large Databases (VLDB 2022). The research paper, titled “Manu: A Cloud Native Vector Database Management System,” explains Manu, the project name for Milvus 2.0, which is an open source vector database created by Zilliz. Xiaomeng Yi, co-author and senior researcher at Zilliz, presented the paper at the conference in an online session on specialized and domain-specific data management.
Zilliz is a primary contributor to Milvus, the vector database which the company says is purpose-built for processing vector data at scale for building AI applications for use cases such as computer vision, NLP, customized search, and new drug discovery. Features of Milvus are integrated into Zilliz Cloud, a fully managed vector database service now available in private preview.
Zilliz says that Manu, or Milvus 2.0, is inspired by the research team’s interaction with 1200+ industry users of Milvus over the past three years. The authors say the main contribution of their VLDB paper lies in the introduction of the real-world demand for vector databases and the design of the basic architecture of a cloud-native vector database. With the development of learning-based embedding models, some vector collections now range in the billions. “Manu relaxes data model and consistency constraints in exchange for the elasticity and evolution necessary for a fully managed and horizontally scalable vector database. Manu is also extensively optimized for performance and usability with hardware-aware implementations and support for complex search semantics,” the company said in a release.
In a blog post, senior researcher Yi explains how Manu, the second major release of Milvus, was designed with specific business requirements in mind, including ever-changing requirements, a more flexible consistency policy, component-level elasticity, and a simpler, more efficient transaction processing model. Gleaned from feedback from the users of the first iteration of Milvus, these requirements were combined with the more common requirements of a distributed system, leading to what Yi calls five broad objectives for Manu: long-term evolvability, tunable consistency, good elasticity, high availability, and high performance.
The research paper explains how Manu supports those objectives through its four-layer architecture which it says enables the decoupling of read from write, stateless from stateful, and storage from computing. There is an access layer, coordinator layer, worker layer, and storage layer, along with a log system as a backbone to connect the decoupled components. Yi explains that this log backbone, a main feature, allows for independent scaling and evolution of each component and facilitates resource allocation and failure isolation.
Another main feature of Manu is its adoption of a delta consistency model which allows for tunable consistency levels. “Delta consistency ensures that the updated data (including inserted and deleted data) can be queried and searched up to delta time units after the data update request is received by Manu,” Yi wrote.
The VLDB review board found that Manu/Milvus 2.0 “is one of the few cloud-native specialized vector databases available in the market” and that the team has introduced “a system for efficient similarity search where the data entities consist of attributes and embedded vectors generated by machine learning models.”
“Zilliz has been investing in frontier research since day one,” said Charles Xie, founder and CEO of Zilliz. “Our dedicated R&D department helps us apply the latest research in Milvus to continuously create value for our users. We are glad that our paper was accepted by VLDB ’22, as the industry has once again recognized our technological leadership. We will continue to seek breakthroughs and apply the latest vector database research results in AI, together with institutions, communities, and ecological partners.”
To read more technical details about Manu, read Yi’s blog post here.