Beyond Naive RAG: HNSW Memory Costs, Hybrid Search and Long Context

Context windows now reach a million tokens and beyond, and that quietly changes what vector search is for. The same shift exposes why high-dimensional indexes struggle at scale, and it points to where enterprise retrieval is going.

The easy years of RAG

When retrieval-augmented generation became the standard pattern for enterprise LLM apps, the recipe looked simple:

1. Split the corpus into snippets of under 1,000 tokens.
2. Run each snippet through an embedding model such as text-embedding-3-large.
3. Keep the dense vectors in a dedicated vector database built on Hierarchical Navigable Small World (HNSW) graphs.
4. At query time, run approximate nearest neighbor (ANN) search and paste the best hits into the prompt.

With 10,000 documents and a 4k context window, that worked well. Then context windows grew past 1M tokens, and the economics shifted. If an engineer can put an entire codebase, a legal archive or a year of filings in the prompt, the case for naive vector search weakens.

Scale adds another problem. Once datasets move from millions to billions of vectors, three hidden costs of pure vector search become visible: exhausted RAM, index quality that degrades as dimensions grow, and semantic recall you can't predict. What follows looks at why vector databases are hitting those limits and what production systems do now.

The HNSW memory problem

Nearly every major vector engine, including Pinecone, Qdrant, Milvus, Weaviate and pgvector, relies on HNSW graphs. HNSW gives fast logarithmic search by arranging points in layers. The upper layers hold long-range links for quick jumps, and the lower layers hold dense local neighbor links.

The catch is that for fast traversal, the graph has to live in RAM.

The arithmetic

Take 1536-dimensional embeddings, as produced by OpenAI's text-embedding-3-small or ada-002, stored as 32-bit floats. Each vector takes 1536 times 4 bytes, which is 6,144 bytes, about 6 KB.

At 100 million vectors:

• Raw vectors: 100 million times 6 KB is roughly 600 GB.
• Graph overhead: linking each node to between 16 and 64 neighbors adds another 20 to 50 percent.
• Total memory: somewhere between 750 GB and 1 TB of fast RAM.

At cloud prices, renting a terabyte of memory only to index text snippets can quickly cost more than the LLM inference itself.

Product Quantization (PQ) and scalar quantization (SQ8) shrink vectors from float32 to int8 or binary forms, but they introduce a second issue: recall drops.

The curse of dimensionality and semantic drift

Above roughly 1000 dimensions, vector spaces show geometric oddities known as the curse of dimensionality. One of them is distance concentration. As dimensions rise, the ratio of the distance to the nearest point over the distance to the farthest point approaches 1.

For cosine similarity, nearly every vector begins to look equally far from every other. Add quantization and the borders between distinct concepts blur further.

Where exact matches fail

Vector search is probabilistic by nature. It measures meaning, not literal tokens. In enterprise systems where literal matches count, that produces serious misses:

• Part numbers and identifiers. A query for "Part #AB-9941-X" can return "Part #AB-9942-X", because both embeddings land in the same cluster.
• Negation and logic. A query like "Contracts without liability caps" often returns contracts that do have liability caps, because the embedding model latches onto the phrase "liability caps".

Long context versus chunks

With models that handle large windows natively, the trade-off between indexing ahead of time and attending directly in context has changed. Cutting a document into 512-token pieces severs cross-references, table dependencies and logical conditions that span the page. When a large window can take whole documents, naive chunking loses its reason to exist, and vector search shifts from finding snippets to routing large blocks of documents.

Hybrid search and the return of BM25

To work around these limits, data engineering is moving away from pure vector stores toward hybrid architectures that combine three pieces:

1. Dense retrieval with vectors, which catches general intent and semantic questions.
2. Sparse retrieval with BM25 or SPLADE, which catches exact keywords, serial numbers and specific entities.
3. Reciprocal Rank Fusion (RRF), which merges both result lists before the top-K candidates reach a reranker model.

A backend pipeline can merge dense and sparse scores with RRF in Python like this:

from typing import List, Dict

def reciprocal_rank_fusion(
    dense_results: List[str],
    sparse_results: List[str],
    k: int = 60
) -> List[Dict[str, float]]:
    """
    Combines dense vector search results and sparse BM25 search results
    using Reciprocal Rank Fusion (RRF).
    """
    rrf_scores: Dict[str, float] = {}

    # Score Dense Results
    for rank, doc_id in enumerate(dense_results):
        if doc_id not in rrf_scores:
            rrf_scores[doc_id] = 0.0
        rrf_scores[doc_id] += 1.0 / (k + (rank + 1))

    # Score Sparse Results (BM25)
    for rank, doc_id in enumerate(sparse_results):
        if doc_id not in rrf_scores:
            rrf_scores[doc_id] = 0.0
        rrf_scores[doc_id] += 1.0 / (k + (rank + 1))

    # Sort documents by descending fusion score
    sorted_docs = sorted(
        rrf_scores.items(), key=lambda item: item[1], reverse=True
    )

    return [{"doc_id": doc, "score": score} for doc, score in sorted_docs]

Why relational databases are winning workloads back

Hybrid needs are pulling workloads toward traditional databases. PostgreSQL (with pgvector and pg_trgm), Elasticsearch and SingleStore let engineers run vector queries next to operational metadata, relational joins and ACID transactions, with no separate vector database to operate.

- PostgreSQL Hybrid Query: Vector Distance + Full Text Match + Relational Filter
SELECT
 id,
 title,
 (ts_rank(text_search_vector, websearch_to_tsquery('english', 'liability clause')) * 0.5) +
 ((1 - (embedding <=> '[0.012, -0.043, …]')) * 0.5) AS hybrid_score
FROM enterprise_documents
WHERE tenant_id = 'org_99412'
 AND status = 'ACTIVE'
ORDER BY hybrid_score DESC
LIMIT 10;

Takeaways

Vector databases aren't going away. What's over is the habit of pushing every document chunk into an in-memory HNSW index without thinking. Four rules for 2026:

1. Don't use vectors alone for exact-match problems. If users search by SKU, name or code syntax, pair vectors with BM25 or an inverted index from the start.
2. Watch the HNSW memory bill. Beyond about 10M vectors, look at disk-backed indexes such as Microsoft's DiskANN, or at binary quantization, before costs run away.
3. Use vectors to route, not to read. Instead of pulling back tiny 200-token chunks, pick the top three to five entire documents, each 50k tokens or more, and feed them to a long-context LLM.
4. Consolidate. Unless you're at multi-billion vector scale with sub-10ms SLAs, your existing relational database, for example PostgreSQL with pgvector, probably does the job and spares you the sync bugs of a distributed system.