What Vector DBMS Can Do – Open Source Tools & Cloud

Epsilla

This – vector DBMSwhich is distributed under the GPL-3.0 license. It can be said that under the hood there is a classic database management system with tables and fields, and vectors are just a separate type of them. Epsilla has a rich ecosystem of integrations, including LangChain And LlamaIndex.

Most vector DBMSs use a structure called the hierarchical small world (HNSW) to find nearest neighbors. However, the performance of this method rapidly degrades when the target accuracy exceeds 95%. According to the Epsilla developers, the reason is that HNSW does not fully utilize the computing resources of multi-core processors. To overcome this limitation, the engineers implemented the SpeedANN indexing algorithm based on parallel graph traversal. Benchmarks show that Epsilla's throughput under real-world load exceeds the capabilities of other DBMSs by almost five times (with a target accuracy level of 95–99%).

Overall, the Epsilla project is quite young, so the existing functionality may not be enough for new users. Although the authors are gradually expanding its capabilities. About a year ago, one of the Epsilla authors noted that the DBMS only supports dense vectors, but they plan to add support for sparse vectors. Interestingly, Epsilla has an experimental feature that allows you to work with the tool as a python library (without docker images). Here is an example of the code that the developers provide:

import epsilla

epsilla.load_db(db_name="db", db_path="/data/epsilla")
epsilla.use_db(db_name="db")
epsilla.create_table(
    table_name="MyTable",
    table_fields=[
        {"name": "ID", "dataType": "INT", "primaryKey": True},
        {"name": "Doc", "dataType": "STRING"},
        {"name": "EmbeddingEuclidean", "dataType": "VECTOR_FLOAT", "dimensions": 4, "metricType": "EUCLIDEAN"}
    ]
)
epsilla.insert(
    table_name="MyTable",
    records=[
        {"ID": 1, "Doc": "Berlin", "EmbeddingEuclidean": [0.05, 0.61, 0.76, 0.74]},
        {"ID": 2, "Doc": "London", "EmbeddingEuclidean": [0.19, 0.81, 0.75, 0.11]},
        {"ID": 3, "Doc": "Moscow", "EmbeddingEuclidean": [0.36, 0.55, 0.47, 0.94]}
    ]
)
(code, response) = epsilla.query(
    table_name="MyTable",
    query_field="EmbeddingEuclidean",
    response_fields=["ID", "Doc", "EmbeddingEuclidean"],
    query_vector=[0.35, 0.55, 0.47, 0.94],
    filter="ID < 6",
    limit=10,
    with_distance=True
)
print(code, response)

Note that the developers offer a cloud platform based on the RAG-as-a-Service model. RAG, or Retrieval Augmented Generation, is an approach to working with large language models, when the system automatically “attaches” additional context from external sources to the user’s request so that LLM can generate the most complete response. In essence, the cloud provides a comprehensive toolkit for managing the entire life cycle of LLM applications: from development and testing to deployment.

MyScale

This – SQL-compatible database management system. It is tailored for developing industrial-grade machine learning models and offers fast vector and text search capabilities. Thus, even developers who are not very familiar with the features of vector databases can quickly get down to business. At the same time, MyScale supports SDK for integration with Python, Node.js, Go, JDBS.

The DBMS is based on the ClickHouse system. The authors explain their choice by the presence of advanced compression mechanisms and support SIMD-commands. You can get started with MyScale using Docker images – instructions are published in GitHub repositories. There is also a link to the official documentation. In it, the developers explain and show how to form SQL tables with a vector index and perform vector search. For example, here is how you can create a table with a vector column:

CREATE TABLE default.wiki_abstract
(
    `id` UInt64,
    `body` String,
    `title` String,
    `url` String,
    `body_vector` Array(Float32),
    CONSTRAINT check_length CHECK length(body_vector) = 384
)
ENGINE = MergeTree
ORDER BY id;

MyScale uses ScaNN, HNSW, IVF nearest neighbor search algorithms, which allows achieving a completeness of 95–99% depending on the search parameters.

Qdrant

This is a universal vector DBMS and API service for working with high-dimensional vectors. Qdrant is written in Rust, which allows the system to work quickly even under serious load. Qdrant uses its own HNSW algorithm and supports the following types of metrics: scalar product, Euclidean and Manhattan distances, and cosine similarity. In addition, the system has advanced filtering, which simplifies working with faceted and semantic search. The authors consider Milvus, Pinecone, and the Dutch Weaviate to be the main competitors of their project.

The DBMS supports integration with Langchain, LlamaIndex, Apache Spark, Apache Airflow and other frameworks. Authors Qdrant provide the ability to test functionality in demo projects. You can evaluate semantic search by text, search by similar images and classification (using e-commerce products as an example).

Those who have already tested this vector DBMS have noted the ease of installation and simplicity of operation. Among Qdrant users there are also large businesses, such as the consulting agency Deloitte. The project is new and actively developing, so it attracts the attention of investors. In January 2024, Qdrant developers received funding in the amount of $28 million, and the total amount of funds they received was $40 million.

But to get started with Qdrant, you don't need ambitious plans – just use Docker and GRPC. It's worth noting that the GRPC API for Qdrant is under development and is disabled by default in production (the GRPC API is for testing and internal use only).

Milvus

Released in 2019, Milvus DBMS is is a flexible platform for analyzing unstructured data. Examples of scenarios include: similarity search of images, videos, audio, and even DNA sequence classification. Most of the vector indexes supported by Milvus use approximate nearest neighbor search (ANNS) methods, including HNSW, FLAT, SCANN, and DiskANN.

The tool's functionality is so complete that the Qdrant team considers Milvus its main competitor in the vector DBMS space. As with Qdrant, the repository contains various demos, such as image search, chatbots, and even molecular search.

One of the components of the computing stack is the Knowhere engine. It allows for horizontal scaling, caching, replication, and failover. You can start working with Milvus using Docker and Kubernetes. The DBMS also provides a Milvus Lite version that runs on all devices (laptops, tablets, mobile phones).

ObjectBox 4.0

ObjectBox — is a lightweight DBMS designed specifically for mobile and embedded devices. The system has minimal technical requirements, so it can be deployed even on a Raspberry Pi. At the same time, the DBMS takes up only a few megabytes of memory (the entire binary file weighs about 3 MB).

Version 4.0 was the first to add vector functionality, and is available for Python, Java, C, and Dart.

Conclusion

Among the most well-known vector search algorithms, the hierarchical small world (HNSW), inverted index (IVF) And DiskANN. Some vendors already offer such functionality in their DBMS. But there are also many open source products on the market that are available for free and allow you to customize the database according to your needs.

Vector databases remain a hot topic in the IT market due to the rapid development of artificial intelligence systems and more. Large players are forming the necessary infrastructure for working with LLM, and small companies and startups are launching automated services based on them.

Cloud elasticity answers requirements of vector databases, which often have to cope with changing workloads and rapidly growing data volumes. The ability of the cloud to dynamically allocate resources ensures efficient scaling without the need for significant upfront investment in physical infrastructure.