Designing RAG for 100 Million Documents: Sharding, Hybrid Search, Cost
What changes when a RAG system grows to billions of chunks: async ingestion, routing, sharding, hybrid retrieval, reranking, authorization, caching and cost control.
This article is published in English.
Getting a RAG demo running takes an afternoon. You split a PDF into chunks, embed them, drop the vectors in a database, pull back the five nearest neighbors, paste them into a prompt and call a model.
That recipe is perfectly fine for a few thousand documents. Once the corpus grows to 100 million documents, it falls apart, and the question you're asking changes. It's no longer "which vector database should we pick?" It becomes: how do you narrow billions of candidate pieces of evidence down to the 10 or 20 passages that matter, without wrecking latency, retrieval quality, availability or budget?
That is a different engineering problem altogether. At this size, RAG is a distributed data and search system with a language model attached at the far end.
What the system has to do
To keep the discussion concrete, assume these requirements:
- More than 100 million documents, in formats such as PDF, HTML and DOCX
- Millions of users asking questions in natural language
- Citations attached to every answer
- Documents that get updated and deleted over time
- Fast retrieval, roughly 1 to 3 seconds before generation starts
- High availability
- Document ingestion that runs asynchronously
Everything reduces to one hard requirement: find the right 10 to 50 pieces of information among potentially billions of chunks.
Start with the arithmetic
The first mistake in large RAG designs is treating a document and a vector as the same thing. Assume:
Documents = 100,000,000
Average chunks/document = 30
Embedding dimensions = 768
Storage/dimension = 4 bytes (float32)
The vector count is then:
100,000,000 × 30
= 3,000,000,000 vectors
Three billion vectors. The raw float storage for them comes to about:
3B × 768 × 4 bytes
≈ 9.2 TB
And those are only the numbers themselves. A real deployment also carries:
ANN index structures
chunk text
document metadata
ACL metadata
IDs
lexical indexes
replicas
snapshots
WALs
temporary indexes
operational headroom
With two replicas of the vector index you're already past 18 TB of raw vectors, before the search structures are counted. If documents average 50 chunks and not 30:
100M documents × 50 chunks
= 5 billion vectors
This is why "we'll put all the chunks in a vector database and run similarity search" doesn't describe a large-scale architecture. It describes where one begins.
Two planes, kept apart
The cleanest way to organize the system is as two independent planes, one for ingestion and one for retrieval. The split matters. A customer uploading 20 million new documents shouldn't push query latency from 300 ms to 8 seconds. Ingestion is tuned for throughput, retrieval for latency, and forcing both through the same pipeline is a reliable way to buy yourself operational pain later.
The vector database is not your source of truth
Keep the original documents in cheap object storage such as S3, GCS or Azure Blob. The vector database is an index, and indexes can be rebuilt. If you move from embedding model E1 to E2 six months from now, you want to regenerate vectors from the stored originals, not try to reconstruct content out of the vector store itself.
Object versioning helps here too, because an update can be recorded as a new version instead of destroying the old one. Amazon S3, for instance, can keep several versions of an object, and lifecycle policies can move older ones to cheaper tiers or expire them.
Ingestion has to be asynchronous
Say a customer uploads 500,000 PDFs. You don't want a POST /documents request that parses, runs OCR, cleans, chunks, embeds, indexes and only then returns a 200. That call could run for hours. Instead the endpoint should do very little:
POST /documents
│
▼
Store document
│
▼
Create ingestion event
│
▼
Return 202 Accepted
Everything else happens off to the side, driven by events. Kafka is one option for the backbone, since topics are split into partitions that consumers can work through in parallel, and ordering holds within each partition, which is handy when events sharing a document or tenant key need a predictable sequence.
The specific tool matters less than the property it provides: ingestion needs a buffer. If your embedding service handles 30,000 chunks a second and suddenly gets work worth 100,000 a second, the system should build a backlog and not fall over.
Make every ingestion step idempotent
At this scale, jobs run twice. Messages get redelivered. Workers die after writing data but before acknowledging the job. Requests time out even though the remote side finished the work. Plan for all of it from the start by deriving deterministic identifiers from stable inputs:
document_id
document_version
chunk_index
embedding_model_version
A chunk ID can then be a hash of the pieces that define it:
hash(
tenant_id +
document_id +
document_version +
chunk_index
)
Receiving the same indexing event twice no longer produces duplicate content, because
upsert(chunk_id, ...)
is safe to retry. Exactly-once execution is hard to achieve. Idempotent execution is usually the more realistic goal.
Chunking is an infrastructure decision
On a small corpus, changing chunk size feels harmless. With billions of vectors it costs real money. Take 100M documents. A strategy that yields 20 chunks per document gives you 2B vectors, while one that yields 60 gives you 6B. A change that looks innocent has just tripled all of these:
embedding compute
vector storage
indexing work
retrieval fan-out
reindexing time
replication traffic
So chunking isn't merely an NLP knob. It's a capacity-planning parameter.
Each chunk should also carry enough metadata to trace it back to its origin:
{
"chunk_id": "c_982...",
"document_id": "doc_51...",
"document_version": 8,
"tenant_id": "tenant_17",
"page": 43,
"section": "Risk Factors",
"language": "en",
"created_at": "...",
"acl_groups": ["finance", "executives"],
"embedding_version": "embed_v4"
}
The text itself may be stored elsewhere, depending on your storage strategy, but the lineage has to survive retrieval. Otherwise citations become painful sooner or later.
Don't search the whole corpus unless you must
This may be the most important idea in the whole design. Take the question "What changed in our European refund policy this quarter?" A naive setup simply searches all 3,000,000,000 vectors. But the question already tells us a lot:
organization = current tenant
region = Europe
topic = refund policy
time = current quarter
permissions = documents user may access
A sound retrieval architecture uses these constraints aggressively, shrinking billions of chunks to a small slice before any similarity search starts. The exact figures depend on the workload, so treat them as illustrations. The principle holds anyway: route first, retrieve second.
Sharding makes the vector layer distributed
At some point one node can't hold the whole searchable corpus, and the index has to be partitioned. Distributed vector engines such as Qdrant scale horizontally by spreading points across shards, with replication placing copies of those shards on several nodes for availability and extra read capacity.
There's a catch, though. If every query must fan out to every shard, adding shards eventually adds coordination work instead of reducing it. The better design makes as many queries routable as possible. Attributes like these can tell you which shards really need to take part:
tenant_id
region
document namespace
language
time partition
This pays off most in multi-tenant systems. Instead of
Query
↓
128 shards
a query might go through
tenant_id = 429
↓
Shard group 18
↓
4 shards
Qdrant's current distributed design, as an example, supports user-defined sharding so operations can target chosen shard keys and avoid touching a whole collection. The lesson applies regardless of vendor: good partitioning cuts query fan-out.
Replication answers a different question
People mix up sharding and replication, but they solve separate problems. Sharding answers how to split a dataset too big for one machine. Replication answers what happens when one of those machines vanishes. Picture this layout:
Shard A → Node 1
Shard B → Node 2
Shard C → Node 3
If Node 2 dies, a slice of the corpus disappears with it. With replication:
Shard A → Node 1 + Node 4
Shard B → Node 2 + Node 5
Shard C → Node 3 + Node 6
the system keeps answering queries when individual nodes fail. Redundancy costs storage, write bandwidth and operational effort, but a production search cluster without it has some uncomfortable failure modes.
Vector search alone falls short
Imagine someone asks, "What does policy HR-48291 say?" Semantic search might return material about HR policies in general, while lexical retrieval is excellent at spotting the exact token HR-48291. Now flip it: "Can employees take extra leave after adopting a child?" Here semantic search earns its keep. The two approaches overlap but cover different ground, which is why a hybrid retrieval stage, running both, is the usual choice.
Elastic's current hybrid-search guidance takes the same route, combining lexical and vector retrieval and suggesting Reciprocal Rank Fusion as one way to merge the rankings. A nice property of RRF is that you don't need to pretend a BM25 score and a cosine similarity are comparable numbers. You combine rank positions instead. In simplified form:
RRF(d) = Σ 1 / (k + rank_i(d))
where rank_i(d) is the position of document d in retrieval system i.
Be careful with sequential prefiltering
There's a tempting shortcut:
3B documents
↓
BM25 returns 10,000
↓
vector search only those
It can work for some workloads, but it can also destroy semantic recall. Picture a user typing "What rules apply to staff who adopt?" and a document that says: "Employees who welcome a child by non-birth placement are entitled to sixteen weeks of family leave." Its keyword overlap with the user's wording is weak. If BM25 throws it out before semantic retrieval ever sees it, no embedding model can rescue it.
So after hard routing constraints, a better habit is generating candidates in parallel from both retrievers and then reranking the merged set. The hard constraints themselves are a separate category:
tenant
permissions
document status
Those should normally be enforced before retrieval starts.
Authorization comes before the LLM sees anything
Picture a corporate corpus holding:
Engineering documents
HR documents
Board documents
Legal documents
Executive compensation
Customer contracts
Semantic relevance is not permission. A CFO's document might be the mathematically closest match to an employee's question, and that still doesn't entitle the employee to read it. The wrong design looks like this:
Vector Search
↓
Retrieve confidential chunk
↓
Send chunk to LLM
↓
Check permission
By the time the permission check runs, the confidential text has already moved into a downstream component. The correct flow resolves who the user is first:
User
↓
Identity
↓
Groups / roles / ACL
↓
Authorized search space
↓
Retrieval
That implies the access-control metadata has to take part in retrieval efficiently. At very large scale, an ACL scheme that needs to scan millions of records after retrieval turns into both a security problem and a performance problem.
Retrieval produces candidates, not context
Say hybrid retrieval yields 400 candidates. Don't place 400 chunks in the prompt. Rerank them first:
Vector candidates: 200
Lexical candidates: 200
│
▼
Fusion
│
▼
~250 unique chunks
│
▼
Reranker
│
▼
top 20–40
The first-stage search is built to find plausibly relevant things quickly. The reranker is built for a different job: given a far smaller set, estimate relevance with more care. This two-stage pattern lets an expensive relevance model work on hundreds of items instead of billions.
Then strip out the redundancy
Even a good reranker can give you something like:
Chunk 1 → page 14
Chunk 2 → page 14
Chunk 3 → page 15
Chunk 4 → page 14
Chunk 5 → page 15
All five might make essentially the same point, and sending every one wastes context tokens. Before building the final prompt, apply some mix of these steps:
deduplication
document diversity
section diversity
near-duplicate detection
adjacent-chunk merging
token budgeting
Sometimes a hit also needs its neighbors. If chunk 47 holds the answer but begins midway through an explanation, the context builder can fetch
chunk 46
chunk 47
chunk 48
instead of hoping vector search will independently retrieve every contiguous piece.
Context construction deserves its own layer
A surprisingly common implementation treats context as nothing more than this:
context = "\n\n".join(retrieved_chunks)
Production systems do better with a dedicated context builder. Its task is to take ranked evidence and produce an efficient prompt payload without losing citation information. Conceptually:
Retrieved Evidence
↓
Deduplicate
↓
Merge related chunks
↓
Enforce token budget
↓
Preserve source IDs
↓
Order evidence
↓
Generate context
The model might then receive something structurally like this:
SOURCE S1
Document: Employee Handbook
Page: 42
Version: 18
Text: ...
SOURCE S2
Document: European Leave Addendum
Page: 7
Version: 4
Text: ...
QUESTION
...
and be told to cite S1, S2 and so on. That's far safer than asking a model to recreate URLs or document identifiers from memory.
Query understanding should be cheap
Some queries benefit from normalization before retrieval. A user types "How much did revenue grow last year in Europe?" Given today's date and the firm's own terminology, retrieval may do better with structured fields extracted from that sentence:
metric = revenue
region = Europe
year = 2025
intent = financial comparison
In a conversation, a follow-up such as "What about Asia?" needs rewriting with the earlier context:
How much did revenue grow in Asia in 2025?
The design point is that these steps rarely need your most expensive reasoning model. Use the cheapest component that does the job dependably. A mature AI stack usually looks like this:
small model
↓
classification
small model
↓
query rewriting
embedding model
↓
retrieval
reranker
↓
relevance
large model
↓
final reasoning
and not like this:
largest model everywhere
Decomposition for multi-hop questions
Some questions can't be answered reliably from a single passage. A question like "which European product dropped the most in revenue, and how did management explain it?" splits into two lookups:
Question 1:
Which European product had the largest decline?
Question 2:
What explanation did management provide for that product?
Retrieval can then collect evidence for each sub-question separately, and the results get joined before generation.
Complex Query
│
▼
Query Decomposer
/ \
/ \
▼ ▼
Subquery A Subquery B
│ │
▼ ▼
Retrieval Retrieval
\ /
\ /
▼ ▼
Evidence Join
│
▼
LLM
At this point large-scale RAG begins to look like a query-planning system more than a simple nearest-neighbor lookup.
Freshness changes how you index
Suppose the corpus holds 100M documents and 2M of them change every day. Rebuilding nightly is probably not what you want, so you need incremental indexing:
Document updated
│
▼
new document version
│
▼
parse changed document
│
▼
generate new chunks
│
▼
embed
│
▼
write new index entries
│
▼
mark previous version inactive
Logical versioning beats immediate destructive replacement. For example, for one document:
document_id = D17
version 41 → status=inactive
version 42 → status=active
Queries filter on
status = active
and old entries can be garbage-collected in the background. From the application's point of view the update commits atomically, with no need for every storage system to change at the same instant.
Version the models as well
Sooner or later you'll switch embedding models, and you should never store vectors without knowing what produced them. At the very least, keep these traceable:
embedding_model
embedding_version
embedding_dimensions
chunking_version
parser_version
Otherwise, six months on, you may find that 15% of an index was built with an older pipeline and have no easy way to say which 15%. A migration can run on parallel indexes:
Documents
│
┌─────────┴─────────┐
▼ ▼
embedding model V3 embedding model V4
│ │
▼ ▼
Index V3 Index V4
│
▼
shadow traffic
│
▼
evaluate
│
▼
switch alias
Don't move billions of vectors and just hope the new retrieval is better. Measure it before switching.
Cache at several levels
Not every query merits the full pipeline. If thousands of employees ask "How many vacation days do we get?", running
embedding
→ distributed retrieval
→ reranking
→ large LLM
every single time is wasteful. A layered set of caches makes more sense:
User Query
│
▼
Exact Response Cache
│ miss
▼
Semantic Cache
│ miss
▼
Retrieval Cache
│ miss
▼
Search
An embedding cache is the obvious companion:
hash(normalized_query)
↓
query embedding
Things get tricky when content changes. An answer cached against
Policy version 17
can turn wrong once
Policy version 18
goes live. The cache key or invalidation scheme must therefore know about corpus and version boundaries. Caching model output without freshness rules is dangerous.
Plan for failure, not only for the happy path
Large systems fail in pieces. Expect things like:
embedding provider unavailable
one vector shard unavailable
search cluster degraded
reranker timeout
LLM rate limited
LLM provider outage
queue backlog growing
parser crashing on malformed PDF
OCR worker running out of memory
Each failure needs an explicit policy. Perhaps:
Vector search fails
↓
Can lexical search answer?
↓
yes
↓
return degraded retrieval mode
or:
Primary LLM unavailable
↓
Fallback model
↓
Lower quality but service continues
or:
Document parser fails 5 times
↓
Dead-letter queue
↓
record failure reason
↓
operator / automated remediation
Retrying forever isn't resilience. Sometimes it's just an infinite loop with a cloud bill attached.
Backpressure
Imagine ingestion normally receives
20M chunks/hour
and embedding capacity can handle
25M chunks/hour
All is well. Then a customer runs a migration and produces
200M chunks/hour
Without backpressure, the failure goes like this:
embedding service overloaded
↓
timeouts
↓
retries
↓
more load
↓
more failures
↓
retry storm
With a buffer in front of the workers:
incoming workload
↓
durable queue
↓
workers consume at safe rate
↓
backlog temporarily grows
Capacity can now be scaled independently of arrival rate. That's another reason the ingestion plane deserves its own architecture.
Observability has to cover retrieval quality
Ordinary monitoring reports things like:
HTTP 200 rate
CPU
memory
disk
p95 latency
error rate
requests/second
All useful, but a RAG service can return HTTP 200 in 800 ms and still give a completely wrong answer. You need AI-specific telemetry as well. A helpful per-request trace might read:
request_id: rq_912
query rewrite:
"parental policy?" → "parental leave policy"
vector retrieval:
200 candidates
145 ms
lexical retrieval:
200 candidates
91 ms
fusion:
278 unique candidates
reranking:
278 → 20
212 ms
context:
13 chunks
8,420 tokens
LLM:
input 9,104 tokens
output 681 tokens
1.8 sec
citations:
3 documents
total:
2.4 sec
Offline evaluation then needs metrics that answer questions such as: was the right evidence retrieved, did it rank high enough, was the answer supported by that evidence, did reranking improve enough queries to justify its latency, and which query classes fail most often? That's more actionable than tracking hallucination as one global percentage.
Evaluate retrieval separately from generation
When a final answer is wrong, there are at least two distinct failure classes. In a retrieval failure, the needed evidence never made it into the context:
Correct document exists
↓
retriever misses it
↓
LLM never sees it
↓
wrong answer
No amount of prompt editing fixes that. In a generation failure, the right evidence did arrive:
correct evidence
↓
context
↓
LLM
↓
incorrect interpretation
Now prompt design, model choice, context organization or the reasoning strategy may matter. If you only score final answers, the two look identical, so a serious evaluation pipeline has to inspect the intermediate stages.
Give latency a budget
"The system needs to be fast" isn't a requirement. Give each stage a budget instead. For illustration, say the target is a first answer in about 2.5 seconds, with a split like this:
API + auth 50 ms
query understanding 100 ms
retrieval 300 ms
fusion 20 ms
reranking 250 ms
context construction 30 ms
LLM time-to-first-token 1,200 ms
network / safety margin 550 ms
-----------------------------------
target ~2,500 ms
These aren't universal numbers. The exercise matters because it shows where time is really going. If retrieval eats 1.8 seconds, switching LLM providers won't touch the main bottleneck. If the LLM accounts for 90% of the delay, spending a month to shave 40 ms off vector search is unlikely to be the best use of effort.
Cost needs the same discipline
A big RAG system has many separate cost centers:
object storage
document parsing
OCR
embedding generation
vector storage
vector replicas
lexical search
metadata database
network transfer
reranking
LLM input tokens
LLM output tokens
observability
backups
So measure cost per useful unit, for example:
cost / 1,000 indexed documents
cost / million chunks
cost / query
cost / successful answer
cost / tenant
The last of those matters most in SaaS. You might discover:
Tenant A
5% of revenue
42% of LLM spend
Without per-tenant metering, all you see is an unexpectedly large monthly infrastructure bill.
Keep expensive storage for what needs it
A helpful mental picture is a gradient from costly and fast to cheap and large:
expensive / fast
▲
│
vector indexes
search indexes
Redis caches
│
metadata DB
│
object storage
│
▼
cheap / large
The full PDF has no reason to sit inside the vector index. Reserve costly storage for what retrieval needs, let object storage hold the source documents, and let the search infrastructure hold searchable representations. Once the corpus runs to hundreds of terabytes, the distinction gets very important.
Hot and cold data
Not all 100M documents get queried equally. Perhaps:
Last 30 days → 70% of queries
Last 12 months → 25%
Older archive → 5%
Treating everything the same wastes infrastructure. One option is a split like this:
Query
│
├────→ Hot index
│
└────→ Archive index when necessary
Time-based routing also stops a question about this week's events from touching ten years of historical shards. Large systems get efficient by avoiding work, not just by doing unnecessary work faster.
Multi-tenancy changes everything
Say your platform serves
10,000 organizations
You almost certainly don't want
10,000 completely independent vector clusters
Yet dumping every tenant into one namespace invites noisy-neighbor and authorization trouble. A workable approach groups tenants by size:
Small tenants
↓
shared shard groups
Medium tenants
↓
partitioned shared infrastructure
Very large tenants
↓
dedicated shard groups
That stops one customer importing 500M chunks from swamping thousands of smaller ones, and it gives a route to isolation for enterprise customers who need it. No single tenant-partitioning strategy suits everyone.
The LLM comes near the end
Look at how much engineering we covered before reaching the model. That's deliberate. A mature RAG request travels like this:
User
↓
API Gateway
↓
Authentication
↓
Authorization
↓
Rate Limiter
↓
Query Understanding
↓
Shard Routing
↓
Hybrid Candidate Retrieval
↓
Rank Fusion
↓
Reranker
↓
Deduplication
↓
Context Builder
↓
LLM
↓
Citation Validation
↓
Response
The LLM matters, but at 100M documents it's one component among many.
The full production picture
Putting it all together, here's the query path, then the ingestion path, then the systems that support both:
CLIENTS
│
▼
┌──────────────┐
│ API Gateway │
└──────┬───────┘
│
┌──────▼───────┐
│ Auth / ACL │
│ Rate Limits │
└──────┬───────┘
│
▼
┌────────────────────┐
│ Query Orchestrator │
└─────────┬──────────┘
│
┌─────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Cache Query Rewrite Metadata
Filters
│ │ │
└─────────────┼──────────────┘
▼
Shard Router
│
┌────────────┴────────────┐
▼ ▼
Lexical Search Vector Search
│ │
└───────────┬─────────────┘
▼
Rank Fusion
│
▼
Reranker
│
▼
Deduplicate
│
▼
Context Builder
│
▼
LLM
│
▼
Citation Validation
│
▼
RESPONSE
================================================================
INGESTION SYSTEM
Documents
│
▼
Object Store
│
▼
Event Queue
│
┌────────────┼────────────┐
▼ ▼ ▼
Parser Parser Parser
│ │ │
└────────────┼────────────┘
▼
Normalization
│
▼
Chunker
│
┌──────────────┴──────────────┐
▼ ▼
Embedding Queue Text Index Queue
│ │
┌─────┼─────┐ ▼
▼ ▼ ▼ Lexical Index
GPU GPU GPU
│ │ │
└─────┼─────┘
▼
Vector Index
================================================================
SUPPORTING SYSTEMS
Metadata DB → document state, versions, ownership
Object Storage → original source documents
Vector Cluster → semantic retrieval
Search Cluster → lexical retrieval
Redis → caches and rate limiting
Message Queue → asynchronous ingestion
Observability → traces, metrics, evaluation
Secrets / IAM → service authorization
Evaluation System → retrieval + answer-quality tests
Anti-patterns to avoid
Reviewing an architecture like this, the following would raise immediate concern:
One giant vector collection with no routing strategy
Synchronous PDF ingestion
Only vector retrieval, no lexical search
ACL filtering after retrieval
No document versioning
No embedding model version
No reranking
Sending top-100 chunks directly to the LLM
Using the LLM for every trivial classification task
No dead-letter queue
No retrieval-level evaluation
No tenant-level cost attribution
Treating the vector DB as permanent document storage
Re-embedding the entire corpus for every small content change
Benchmarking only average latency instead of tail latency
Any one of them can pass in a prototype. Together, at billion-vector scale, they become extremely painful.
The core principle
The most useful way to think about large RAG isn't this:
question
→ vector search
→ LLM
It's this:
billions of possible evidence units
↓
routing constraints
↓
relevant partitions
↓
cheap candidate search
↓
hundreds of items
↓
expensive reranking
↓
tens of passages
↓
context optimization
↓
LLM
↓
grounded answer
The system is a funnel. At each stage we spend more computation on fewer candidates:
Cheap operation → huge search space
Moderate operation → smaller candidate set
Expensive operation → tiny candidate set
Most expensive model → final context only
That idea is what keeps the architecture economically viable.
What to take away
A RAG system over a few thousand documents is mostly an application-engineering exercise. Over 100 million documents, it turns into search infrastructure, distributed systems, data engineering, security and cost control, all demanding attention at the same time. The vector database matters, and so do the embedding model and the LLM, but the architecture around them matters more than most tutorials admit.
The mental model that scales: store the source cheaply, process asynchronously, version everything, partition on purpose, enforce authorization early, retrieve with several signals, rerank hard, give the model only high-quality evidence and measure each stage on its own. Once the corpus reaches billions of chunks, the best optimization is no longer making search faster. It's arranging things so the bulk of the system never touches the bulk of the corpus.