Inside Vector Databases for RAG: Chunk Storage, HNSW, IVF and Re-ranking

A retrieval-augmented generation system has to answer an early practical question: where do the text chunks live? The most common answer, and the one most production systems use, is to put each chunk inside the vector database right next to its embedding.

What a stored record looks like

In setups built on FAISS, Pinecone or Weaviate, every record bundles three things: the embedding vector, the chunk text and some metadata.

{
        "embedding": [0.123, 0.456, ...],
        "text": "Transformer models are powerful...",
        "metadata": {
        "doc_id": "doc1",
        "page": 5
        }
}

Keeping the chunk and its vector together has three payoffs. Search is fast because the matching text comes back in the same response. The architecture stays simple, with no second store to query. And latency stays low, which matters when an LLM is waiting on the result in real time.

Retrieval then runs in five steps. A user asks a question, the question is turned into an embedding, the database finds the nearest stored embeddings, the matching chunk text is returned directly, and the LLM writes its answer from that text. There's no extra lookup anywhere.

How the search stays fast

The usual one-line description is that a vector database finds similar embeddings using cosine similarity, dot product or Euclidean distance. That's accurate, but it skips the interesting engineering. With millions or billions of vectors, how does a query come back in milliseconds?

The naive route is brute force: compare the query against every stored vector. It gives perfect accuracy and awful speed. A million vectors means a million comparisons for each query, and that doesn't scale.

Real systems use approximate nearest neighbor (ANN) search. The idea is to shrink the search space first and then examine only a small subset of vectors, accepting a little lost accuracy in return for a big speedup. ANN is delivered through specialized index structures. FAISS, for example, ships indexes of this kind, and both HNSW and IVF are available there.

HNSW, a layered graph

HNSW stands for Hierarchical Navigable Small World. It's graph-based, and Pinecone and Weaviate use it internally. Each vector becomes a node, and each node links to similar neighbors. A search is a walk across that graph.

The graph has several layers. The top layer holds very few nodes and gives a coarse view, while the bottom layer holds every node in detail. A highway versus local streets comparison works well here.

Building it goes like this. When an embedding is inserted, it's given a random level, so most nodes land only on the bottom layer and a rare few reach the top. Within each level the node connects to its nearest neighbors, measured by cosine, dot product or Euclidean distance. What you end up with is a network where close vectors are tightly connected and distant ones only loosely.

Searching starts at the top layer. The query finds its best-matching node there, drops into the next layer from that node and repeats the greedy hop. This continues down to the bottom. In the upper and middle layers the search is strictly greedy: move to the best neighbor and stop when nothing improves, then descend. The bottom layer is more exploratory. A priority queue of candidates starts from the entry node, and the search keeps adding good matches and examining several neighbors at once.

Two knobs matter in practice:

• M is the number of connections per node. A larger M raises recall but costs more memory and build time.
• efSearch is how many candidates get explored during a query. A larger value improves recall at the price of higher latency.

Put simply, M shapes the quality of the index, while efSearch trades accuracy against speed at query time.

IVF, clustering first

IVF (inverted file index) is common in FAISS and takes a completely different approach. It doesn't build a graph. It clusters, grouping similar vectors and searching only the relevant groups. A library is a good picture: books are sorted by category, so nobody scans every shelf.

On insertion, a clustering algorithm such as k-means splits the vectors into N clusters. Each cluster has a centroid, and every vector is assigned to its nearest one. At query time, the embedding is compared with all the centroids, the closest cluster is chosen, and the query is scored against each vector inside that cluster to return the top results.

The two methods differ in memory. IVF stores fewer connections, so it uses less. HNSW keeps a graph with many edges per node, so it uses more.

Where pure vector search falls short

Embeddings capture meaning, context and conceptual closeness. They're weaker with exact keywords, numbers and IDs. Search for "kafka error code 504" and a vector index might return "kafka network issues" or "timeouts in distributed systems" while missing the document that actually contains 504.

Production RAG systems answer this with hybrid search, which pairs semantic retrieval (embeddings plus ANN such as HNSW or IVF) with keyword retrieval for exact matches using BM25. In a typical architecture the two searches run on separate systems, and their results are merged and re-ranked.

More modern vector databases handle hybrid search internally, so a second system isn't needed. Weaviate and Pinecone both support it natively. It's quickly turning into the default for production, since it connects understanding of meaning with matching of exact intent.

Doesn't hybrid mean storing everything twice?

With an external setup, yes: the same chunks often sit in both the vector database and a keyword engine such as Elasticsearch. That's deliberate. The vector store keeps an HNSW or IVF index for semantic similarity, whereas the keyword engine keeps an inverted index for exact matching. Each index is built for its own purpose, so this isn't redundancy.

Teams typically keep a single source of truth, like a database or data lake, and sync it into both systems through pipelines. That keeps the two consistent while retrieval stays fast and accurate.

Re-ranking the top results

Plenty of engineers assume the job is finished once the top-K comes back from the vector database. It isn't. Those results are approximately relevant, not perfectly ordered, and fine-grained relevance can be missed.

Take the query "kafka error 504 fix". The top-K might return "Kafka timeout issues", then "Distributed system errors", then "Error code 504 gateway timeout". The third result should plainly be first, and vector search won't always manage that.

Re-ranking takes the retrieved chunks and reorders them using a deeper reading of the query. Vector databases generally don't do this themselves, so the logic lives in your application layer. Two approaches dominate:

1. LLM-based re-ranking. Send the query and the chunks to an LLM and ask it to rank them by relevance. It's easy to implement and the model understands context well. On the downside, it's slow because of the extra call and it gets expensive at scale.
2. Cross-encoder. This is the standard for high-quality re-ranking. The model takes the query and one chunk together as a pair and outputs a relevance score. You repeat that for all top-K candidates and sort by score. It's deterministic and considerably more accurate.

A compact cross-encoder example with LangChain:

"""Super-simple CrossEncoder reranking example.

Steps:
1. Define a query and a few short documents.
2. Build (query, doc) pairs.
3. Use a CrossEncoder to get a relevance score for each pair.
4. Print raw scores, then print documents sorted by score.
"""

from sentence_transformers import CrossEncoder


def main() -> None:
        # 1. Create model
model_name = "cross-encoder/ms-marco-MiniLM-L-6-v2"
print(f"Loading CrossEncoder model: {model_name}\n")
model = CrossEncoder(model_name)

    # 2. Query and documents
        query = "What is a vector database?"
documents = [
        "A vector database stores embeddings and allows similarity search.",
        "Relational databases store structured data in tables.",
        "FAISS is a library for efficient similarity search of vectors.",
        "Vector databases are used in AI applications like RAG.",
        ]

        # 3. Build (query, doc) pairs
pairs = [(query, doc) for doc in documents]

        # 4. Get scores
scores = model.predict(pairs)

print("Query:\n  " + query + "\n")
print("Raw scores (higher = more relevant):")
    for doc, score in zip(documents, scores):
print(f"  score={score:.4f}  |  doc={doc}")

    # 5. Sort by score (descending)
ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)

print("\nDocuments sorted by cross-encoder score:\n")
    for rank, (doc, score) in enumerate(ranked, start=1):
print(f"Rank {rank}: score={score:.4f}")
print(f"  {doc}\n")


if __name__ == "__main__":
main()

Output
****************************************************
Query:
  What is a vector database?

Raw scores (higher = more relevant):
  score=8.4591  |  doc=A vector database stores embeddings and allows similarity search.
  score=-7.1590  |  doc=Relational databases store structured data in tables.
  score=1.0106  |  doc=FAISS is a library for efficient similarity search of vectors.
  score=7.0736  |  doc=Vector databases are used in AI applications like RAG.

Documents sorted by cross-encoder score:

Rank 1: score=8.4591
  A vector database stores embeddings and allows similarity search.

Rank 2: score=7.0736
  Vector databases are used in AI applications like RAG.

Rank 3: score=1.0106
  FAISS is a library for efficient similarity search of vectors.

Rank 4: score=-7.1590
  Relational databases store structured data in tables.

Measuring similarity

Retrieval quality depends quietly on how vector closeness is computed. Three measures cover almost everything.

Cosine similarity looks at the angle between two vectors, in other words direction only. Two arrows pointing the same way are highly similar, and opposite arrows are completely different, regardless of length. The formula is cos(θ) = A·B / (‖A‖‖B‖). With the query "Kafka error 504", a document about "Kafka error handling" would score high (around 0.9) and one on "Database indexing" low (around 0.2). It suits embeddings and is the usual choice in RAG, since meaning matters and magnitude doesn't. Try query = [1, 1], docA = [2, 2] and docB = [1, 0]: docA points the same way as the query, so its cosine is a perfect match even though it's longer.

Dot product blends direction and magnitude, A·B = |A| × |B| × cos(θ). Bigger vectors count as more important, and vectors pointing the same way count as more similar.

Euclidean distance measures the straight-line gap between points. Closer means more similar. It's used mostly for clustering problems.

How the pieces fit

A good RAG pipeline is more than a vector database bolted onto an LLM. Direction in an embedding carries meaning and magnitude carries signal strength. FAISS, Pinecone and Weaviate use HNSW and IVF to retrieve at scale. After that, quality depends on a few more decisions:

• chunk documents carefully so context is preserved
• use hybrid search to cover semantic and keyword matches
• re-rank with a cross-encoder or an LLM
• write strong prompts to steer the model and curb hallucination
• tune top-K and score thresholds

The whole path reads: data, chunking, embeddings, indexing, retrieval, re-ranking, prompting, LLM, answer.