Choosing a Vector Database for RAG: Eight Options Compared

Picking a vector database for RAG comes down to a few honest questions about size, filtering, tenancy and how much operations work your team can absorb. Here's the short version before the details. Chroma and pgvector fit small projects. Qdrant and Milvus are the usual picks for self-hosted production. Pinecone is the fully managed option. If Postgres is already in your stack, pgvector costs nothing extra, and if you just want a local setup running in minutes, Chroma installs with a single pip command.

Open source engines such as Qdrant, Milvus and Weaviate hold their own against paid platforms on recall and latency, and each has a free self-hosted route next to an optional paid cloud. One more number worth remembering: combining dense vector retrieval with sparse BM25 keyword search has lifted Recall@10 from roughly 78% to roughly 91% in production benchmarks, which is why hybrid search shows up in so many of the options below.

Vector databases are the retrieval layer behind most generative AI apps. They let a company ground a language model in its own data so answers stay accurate and specific to the domain. As the pile of unstructured data grows, retrieval becomes the main bottleneck for quality, and choosing well means understanding indexing algorithms, filtering behavior and what the infrastructure will cost.

What a vector database actually does

A vector database holds high-dimensional embeddings and ranks stored rows by how close they sit to a query vector, using cosine similarity, dot product or L2 distance. A relational engine matches exact tokens, and BM25 scores term overlap. A vector search can return documents about "affordable cars" for the query "cheap automobile," because the two phrases land near each other in embedding space.

There's a catch. Exact k-nearest-neighbor search costs O(N·d) per query, which stops being practical somewhere past a few hundred thousand vectors. Vector databases get around that by accepting a small loss in recall in exchange for approximate nearest-neighbor (ANN) indexes that answer in roughly logarithmic time.

What to measure before choosing

Teams that skip evaluation tend to meet the bottleneck in production. Four areas deserve a look.

Retrieval quality and latency. Recall@K tells you how often the system finds the truly best matches inside its top K results. Higher recall means the model receives the best information on offer. Latency, in milliseconds, matters equally, and engineers watch p95 or p99 because those reflect the slow end of what users experience.

Metadata filtering. Often a query has to be narrowed by structured attributes. A search for "company policy" might need to cover only documents after a given date or for one department. Good filtering keeps irrelevant or unauthorized records out of the results. Databases with pre-filtering are generally better, since they discard unwanted records before the vector search starts, which protects accuracy.

Ecosystem fit. A database becomes more useful when it plays well with the rest of the AI stack. Connectors for LangChain and LlamaIndex shorten the time to a working RAG pipeline. Client libraries in several languages and support for AWS, GCP and other clouds decide how smoothly the thing slots into existing enterprise software.

Operational readiness. This covers whether the system can survive a demanding production setting:

• Scalability, meaning billions of vectors and heavy query traffic.
• Isolation between users in multi-tenant apps.
• Compliance with standards such as SOC 2, HIPAA and GDPR.
• Observability, so search performance and resource use can be monitored.

Free and open-source options

Open source engines let you run your own infrastructure and skip recurring subscription fees. The engines below are listed roughly from lightest to heaviest.

Chroma

Chroma is an embedding database built for developer speed. It's the shortest route from an empty repository to a working RAG prototype, and many people use it as their first vector store. It suits hackathons, local development and quick experiments where ease matters more than distributed scale.

What it offers: a small Python API that moves from throwaway local tests to a persistent client-server mode, local storage backed by SQLite so no database server has to be provisioned, and built-in embedding functions that turn raw text into vectors on insert with defaults like all-MiniLM-L6-v2. It connects to LangChain, LlamaIndex, OpenAI and Hugging Face.

On the plus side, installation is a plain pip install chromadb, text-to-vector conversion happens without extra code, and running locally gives full control over data and infrastructure. On the minus side, it lacks the security and multi-tenancy features strict enterprise environments need, and it can't keep up with purpose-built distributed systems once data passes a few million vectors.

The core database is free under Apache 2.0, and a hosted cloud version now exists for production scaling.

Milvus

Milvus is a cloud-native, highly distributed vector database that separates compute from storage. It targets organizations handling extreme data volumes, with hundreds of millions to billions of vectors, where scaling out is mandatory and not optional.

Its architecture splits data nodes, query nodes and index nodes. It supports many index types, including HNSW, IVF and DiskANN, so you can tune for particular hardware, and it has serious GPU-accelerated indexing and search. Nodes can be added or removed on the fly without downtime.

Strengths: billion-scale datasets behave predictably without bad p99 latency spikes, strict node isolation lets compute go exactly where a workload needs it, and the open-source community is large, with plenty of documentation and plugins. Weaknesses: deploying, tuning and running the Kubernetes-based system takes real engineering effort, and for a simple RAG app the extra moving parts, such as etcd and Kafka, are unnecessary overhead.

The engine is free under Apache 2.0. Zilliz, the company behind it, sells managed hosting starting at $155 per month.

Qdrant

Qdrant is an open-source vector search engine written in Rust, and its priorities are memory safety, throughput and very good payload filtering. Teams that want speed and reliability tend to land here, particularly for RAG with rich metadata filters, custom scoring and tight latency budgets.

Under the hood is a Rust storage engine called Gridstore, tuned for vector workloads. A composable search design lets one query mix dense vectors, sparse vectors and metadata filters. Filtering is applied inside the graph during HNSW traversal, which avoids the recall drops that come with filtering after the search. There's also Qdrant Edge, a lightweight embedded build for mobile and robotics.

Typical p50 latency runs around 4 to 5 ms, filtering stays accurate under tight constraints, and the license (Apache 2.0) is permissive. The trade-offs: contributing to the Rust core means a steeper learning curve, and getting the best HNSW behavior requires careful memory and index settings.

The open-source release is free. Qdrant Cloud has a free managed tier with 1 GB of storage, plus paid usage-based clusters.

Weaviate

Weaviate is an AI-native open-source database organized around graph-like objects, blending several search styles and data types into one query experience. It's a fit when you need mature hybrid search, complex relationships between objects and an AI workflow built into the database.

It blends dense similarity and BM25 keyword ranking natively. Vectorization modules connect to OpenAI, Cohere and Hugging Face and embed text for you. Data lives in a class-based schema geared to relationships, and multi-user isolation and replication are included.

The benefits are less boilerplate, since text-to-vector conversion happens inside the database, tunable hybrid parameters that balance exact keyword precision against broad semantic recall, and stable millisecond responses up to hundreds of millions of objects. The costs are the initial friction of adapting to its class schema and a heavier footprint than embedded options.

Software is free under BSD-3. Weaviate Cloud offers serverless managed plans from $45 per month.

pgvector for PostgreSQL

The pgvector extension turns PostgreSQL into a capable vector store, keeping embeddings beside ordinary transactional data. It's the natural choice for teams already committed to Postgres who'd rather not stand up another database.

It adds a native vector column type to regular SQL schemas, supports cosine similarity, L2 distance and inner product, builds HNSW and IVFFlat indexes, and runs similarity queries inside standard ACID transactions.

The big win is skipping the operational overhead, security reviews and upkeep of a separate system. Existing Postgres monitoring, backup and replication carry over, and metadata filtering is just a familiar WHERE clause beside a distance function. The downsides are higher memory use and weaker scaling than engines built only for vectors, and noticeable latency growth beyond about 50 million vectors unless something like pgvectorscale is added.

The extension is free and open source. What you spend depends on how you host Postgres.

Paid and managed options

Managed databases hand you a low-maintenance experience so the team can focus on the application instead of servers.

Pinecone

Pinecone more or less defined the managed vector database category with a serverless, zero-ops design that hides all backend infrastructure. It's meant for teams that want to reach production quickly without managing servers, clusters or index parameters.

It runs purely as a cloud SaaS, splits read, write and storage layers so each scales independently, hosts embedding and reranking models through Pinecone Inference, and supports dense vectors, sparse vectors and effective metadata filtering. In return you get no infrastructure chores, no patching or index rebuilds, wide framework and cloud compatibility, and high availability with predictable performance under load.

The drawbacks are that it's closed-source, which means real vendor lock-in, and that consumption pricing climbs fast for workloads with heavy reads and writes.

Pricing as listed: a free Starter tier (2 GB, 1M reads, AWS us-east-1 only); Standard with a $50 monthly minimum at $0.33 per GB-month plus roughly $4 to $4.50 per million write units and $16 to $18 per million read units; Enterprise with a $500 monthly minimum; and a HIPAA add-on at $190 per month.

Turbopuffer

Turbopuffer is a serverless search engine built straight on cloud object storage, and cost efficiency is the whole point. It's the strongest candidate for multi-tenant SaaS and AI products with huge numbers of namespaces where spend has to stay in check.

It skips conventional disk and searches directly over object storage, handles billions of vectors with no hard cap on namespaces, uses an SPFresh index tuned for search, and makes writes visible in results immediately. Costs for storage and queries drop sharply compared with older managed services, per-pod pricing traps that punish multi-tenant designs are avoided, and SOC 2 and HIPAA readiness come on standard plans.

Against that, it's closed-source with a shorter track record than established rivals, and its plugin ecosystem and integrations are thinner. Billing is purely usage-based with a $64 monthly minimum on standard plans, and big namespaces get volume discounts on queries.

TiDB Vector Search

PingCAP's TiDB Vector Search pairs a distributed SQL database with its own embedding engine, so large transactional workloads and semantic retrieval run in one scale-out system. It makes sense when you need very large horizontal scale along with complex SQL filtering and vector search on the same data at once.

The platform combines embedding storage and distributed SQL, builds regular SQL indexes next to HNSW vector indexes, spreads queries across nodes for high availability, and supports hybrid search and exact keyword matching alongside vector similarity. That removes sync problems and pipelines between an application database and a separate vector store, allows heavily filtered multi-tenant retrieval through SQL joins, and scales out without choking queries.

It's overkill for a lightweight, vectors-only RAG pipeline, and tuning performance fully means understanding distributed SQL in depth.

On TiDB Cloud Starter the free tier is the most generous in the category: 5 GiB of row storage, 5 GiB of columnar storage and 50M request units per cluster, with up to 5 clusters per organization. That adds up to 25 GiB row, 25 GiB column and 250M RUs a month at no charge. Beyond it, you pay $0.10 per million RUs and $0.20 per GiB-month for row storage in us-east.

Where vector RAG falls short

Vector search is powerful and still has weaknesses that hurt accuracy in enterprise use.

• Lost context. Standard vector RAG cuts documents into small chunks to fit context windows. That mechanical step wipes out the structure of complicated documents, and a small chunk easily loses its link to the headers or sections that gave it meaning.
• Imprecision. Retrieval depends on closeness in vector space, so lookalike terms or product names with shared wording get confused. Wrong facts end up in the prompt, and hallucinations follow.
• Costly updates. Switching to a better embedding model means re-embedding the whole corpus and rebuilding large indexes from scratch, which drives up compute and spending.

Graph RAG is one alternative. It models data points as nodes and relationships as edges, and tools such as the Writer Knowledge Graph keep explicit links between documents. That supports multi-hop reasoning, cuts hallucinations and lets a team update a single fact without re-embedding everything.

A practical way to decide

Begin with what you already run. On Postgres with under roughly 10M vectors, pgvector handles the job without a new dependency, and pgvectorscale extends it to 50M and beyond. On MongoDB, use Atlas Vector Search. On Elastic or OpenSearch, stay put, since OpenSearch 3.0 closed most of the gap. On Redis, use Redis 8's native vector support with FT.HYBRID.

Starting from scratch, weigh three things:

1. Scale and operations capacity. Prototyping under 1M vectors points to Chroma. Self-hosted production with strong filter performance points to Qdrant. Billion-scale or GPU-accelerated work points to Milvus. Managed production with compliance needs points to Pinecone or Weaviate Cloud.
2. Shape of the cost. For heavy multi-tenant fan-out where most tenants are idle, Turbopuffer's 10 to 100 times savings are real, though cold queries take around 500 ms. For cost-sensitive setups above 50M vectors with predictable load, pgvector plus pgvectorscale fits.
3. Reasoning requirements. For multi-hop questions, explainability or relational structure, add a knowledge graph such as Neo4j, Memgraph or FalkorDB and run Graph RAG alongside vectors, not in place of them.

Common questions

How do you build a vector database for RAG? Split the documents into chunks, create embeddings with a model such as an OpenAI or BGE one, then store and index the vectors in something like Qdrant, Milvus or pgvector. For a fully local setup, Chroma or FAISS run on one machine with no cloud involved.

What role does it play? It's the retrieval engine sitting between your documents and the LLM. It stores embeddings and, at query time, returns the most relevant chunks so the model answers from grounded context and not from memory alone.

Which database should you use? It depends on scale and stack. Chroma and pgvector are the best free picks for small to mid-sized work, Qdrant and Milvus lead for self-hosted production, and Pinecone is the top managed pick when you want zero ops.

How are vectors used in the pipeline? An embedding model converts text chunks into high-dimensional vectors. The database stores them, and when a question arrives it finds the nearest vectors to the query vector through ANN search and passes those chunks into the prompt.

Is a vector database always required? No, even though it remains the industry default. Vectorless systems such as PageIndex skip embeddings entirely. They turn documents into hierarchical trees and let the LLM reason over the table of contents to find answers. Many enterprise teams also run graph databases for GraphRAG, retrieving context through explicit connections instead of spatial similarity.

What are the seven types of RAG? Naive RAG (basic retrieve then generate), retrieve-and-rerank (the usual production approach), hybrid RAG (dense and sparse search together), multimodal RAG (text, images and audio in one pipeline), Graph RAG (knowledge-graph traversal for relational queries), agentic RAG (an LLM agent picks the best retrieval strategy per query) and HyDE RAG (the model drafts a hypothetical answer first, then retrieves against it).