Query-Time RAG at Ten Million Documents: Funnels, Routers, Agents and Evals

Getting ten million documents into a vector database is only half the work. The first half of this topic dealt with ingestion: document routers such as Docling, Unstructured and Apache Tika, layout-aware chunking, delta indexing with MinHash, and a dual-storage design that indexes real-world files without corrupting data or inflating the bill.

The second half is query time. Millions of embedded chunks accomplish nothing if retrieval takes eight seconds, chokes on specific error codes, exposes confidential payroll data, or swallows a prompt injection hidden in some legacy file. What follows covers a retrieval funnel that answers in under 200 ms, smart query triage, agentic orchestration, continuous evaluation and a complete script.

1. A multi-stage retrieval funnel

A typical tutorial says to embed the user query and run a top_k=5 similarity search over the entire database. At ten million documents that approach creates two big problems.

1. Dense embeddings capture abstract meaning well but fail on exact keywords, alphanumeric part numbers and error codes. A search for Error 504 Gateway Timeout may return general timeout discussions and not the precise troubleshooting page.
2. Scoring every candidate with a heavyweight relevance model is far too slow for interactive use.

Production systems therefore treat retrieval as a funnel that narrows candidates step by step:

[ 10,000,000 Total Document Chunks ]
                 │
                 ▼
     [ Step 1: SQL Pre-Filter ] ────────► Filter by Tenant / Dept / Role / Region
                 │
                 ▼
       [ ~50,000 Candidates ]
                 │
                 ▼
     [ Step 2: Hybrid Search ] ─────────► Dense Vectors (Qdrant) + Sparse BM25
                 │
                 ▼
        [ Top 100 Candidates ]
                 │
                 ▼
       [ Step 3: Cross-Encoder ] ───────► Cohere Rerank / BGE-Reranker
                 │
                 ▼
       [ Final Top 5 Chunks ] ──────────► Passed to LLM Context Window

Stage 1: relational pre-filtering

Before any high-dimensional math, metadata filters on user role, department or document status shrink the search space. Dropping from ten million chunks to perhaps 50,000 cuts latency and enforces access control in the same move:

from qdrant_client.models import Filter, FieldCondition, MatchValue

# Restrict search space by user session permissions before distance scoring
user_access_filter = Filter(
    must=[
        FieldCondition(key="department", match=MatchValue(value="Engineering")),
        FieldCondition(key="is_active", match=MatchValue(value=True))
    ]
)

Stage 2: hybrid search

Real searches come in two complementary flavors, and it helps to understand both before looking at the fusion algorithm.

Dense or semantic search turns text into embeddings in the vector database, placing words with similar meanings close together. It copes well with paraphrases, synonyms and conversational phrasing, though it can miss an exact keyword and return documents that merely share a concept.

Sparse or keyword search, with algorithms like BM25 or the older TF-IDF, scores a term by how often it occurs in a document relative to how rare it is across the corpus. It catches exact words reliably but knows nothing about meaning.

The tempting answer is to run both and add their scores, which is where Reciprocal Rank Fusion comes in. RRF merges several ranked lists into one unified list with no need to normalize raw scores. A document that places near the top of both rankings floats to the top of the combined list.

Stage 3: cross-encoder reranking

Hybrid search quickly yields about a hundred candidates, but ordering them purely by vector distance can still surface irrelevant ones. Two ways of encoding explain the fix.

A bi-encoder is how ordinary vector search works. Documents become vectors during ingestion, the user's prompt becomes another vector at query time, and simple geometry such as cosine similarity or a dot product compares them. This is fast but only moderately accurate.

A cross-encoder reads the query and a document together and outputs a score for how relevant that document is to that query. It is thorough and therefore slow, since scoring a million documents for one search would take hours. The answer is to combine them. The bi-encoder finds the top 100 candidates, and the cross-encoder then reorders only those into the best ten, which takes under 100 ms.

2. A conditional query router

Most chat interfaces are a plain text box, so users can type anything. A message such as "summarize our conversation so far" or "translate the results to German" doesn't need a document lookup at all, and sending it through the full pipeline wastes time and money.

A conditional router sits at the entrance as a quick classifier that decides where each message goes:

User Query
    │
    ▼
[ Intent Classifier / Router ]
    │
    ├── Simple Math / Logic ──────────► Direct Calculator / Python REPL
    ├── Conversational / Follow-up ───► Direct LLM Memory Context
    └── Domain Knowledge Request ─────► Full Hybrid RAG Pipeline

Conceptually, a small prompt asks a model to classify the query into one of a few routes, such as retrieval from internal documentation or a direct answer:

# Conceptual Router Pattern
def route_query(user_query: str) -> str:
    prompt = f"""Classify the user query into one of these routes:
    - RETRIEVE: Needs internal company documentation/database lookup.
    - COMPUTE: Pure math, calculation, or logic.
    - DIRECT: Conversational, greetings, or basic language rewrites.

    Query: {user_query}
    Classification:"""

    # Run a fast, lightweight classifier (or small local SLM)
    decision = fast_classifier(prompt).strip()
    return decision

3. Beyond a single lookup: orchestration and feedback

Many real questions need several reasoning steps. Consider a request to compare 2025 APAC sales figures against the 2024 compliance guidelines and flag discrepancies. It splits into three sub-tasks: find and extract the 2025 APAC sales data, retrieve the 2024 compliance guidelines, and look for conflicts between them.

A single-shot system fails here because it tries to answer everything in one prompt. Agentic RAG addresses this through orchestration frameworks like LangGraph or CrewAI, which split the problem into coordinated specialist tasks. A planner can spawn several agents, each using the same retrieval pipeline to gather its slice of evidence:

                  [ Orchestrator / Planner ]
                              │
            ┌─────────────────┴─────────────────┐
            ▼                                   ▼
    [ Researcher Agent ]                [ Compliance Agent ]
    • Retrieves 2025 Sales Data         • Retrieves 2024 Regulations
    • Extracts regional tables          • Parses policy constraints
            │                                   │
            └─────────────────┬─────────────────┘
                              ▼
                      [ Synthesis Agent ]
                      • Reconciles numbers
                      • Validates output consistency
                              │
                    Confidence Score Check
                       │             │
              [ Low Confidence ]     [ High Confidence ]
                       │                     │
                       ▼                     ▼
              Loop back & refine     Final Guardrail Validation

Self-correction

When retrieval returns chunks with low similarity scores, or when the synthesizing agent notices a contradiction, the agent should react. It rejects the weak text, rewrites the query by expanding abbreviations or trying synonyms, and queries the retrieval engine again, instead of returning an invented generic answer. Agents orchestrate and decide, yet without a solid RAG pipeline underneath they have nothing to work with.

4. Continuous evaluation

A production system needs ongoing scoring through automated frameworks such as Ragas, TruLens or DeepEval. They look at several dimensions:

┌──► Faithfulness (Is the answer grounded in the retrieved chunks?)
RAG Evaluation ─┼──► Answer Relevance (Did it actually answer the user's prompt?)
                ├──► Context Recall (Did retrieval find all necessary reference chunks?)
                └──► System Latency & Token Cost (Is it cost-effective at scale?)

Beyond operational metrics, several service level indicators gauge how well retrieval is working:

• Faithfulness is the share of claims in the answer that the retrieved context supports, which works as a hallucination check.
• Context precision is the share of retrieved chunks, among all those placed in the prompt, that are actually relevant.
• Context recall asks whether every ground-truth fact needed for the question was retrieved.
• Latency, both end to end and per stage, covers router, filter, hybrid search, reranker and generation.

Each indicator deserves a target baseline so that quality and performance agreements can be checked.

5. A complete retrieval and generation script

The Python script below covers the execution side. It applies metadata pre-filtering, retrieves vectors from Qdrant, assembles context and calls a local LM Studio server for inference:

from openai import OpenAI
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue

# 1. Initialize Clients
# Pointing to local LM Studio running on port 8080
ai_client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="lm-studio")
qdrant_client = QdrantClient(url="http://localhost:6333")
COLLECTION_NAME = "enterprise_knowledge_base"
EMBEDDING_MODEL = "nomic-ai/nomic-embed-text-v1.5"
def retrieve_and_generate(user_query: str, user_department: str) -> str:
    print(f"\n🔍 Processing query: \"{user_query}\" for department: [{user_department}]")

    # 2. Vectorize the User Query
    query_resp = ai_client.embeddings.create(
        input=[user_query],
        model=EMBEDDING_MODEL
    )
    query_vector = query_resp.data[0].embedding
    # 3. Stage 1 & 2: SQL Pre-Filter + Vector Search
    # Filter by user department and active document status
    access_filter = Filter(
        must=[
            FieldCondition(key="department", match=MatchValue(value=user_department))
        ]
    )
    search_results = qdrant_client.search(
        collection_name=COLLECTION_NAME,
        query_vector=query_vector,
        query_filter=access_filter,
        limit=3
    )
    if not search_results:
        return "No relevant or authorized documents found."
    # 4. Context Assembly with Breadcrumbs
    context_blocks = []
    for hit in search_results:
        breadcrumb = hit.payload.get("breadcrumb", "General")
        text = hit.payload.get("text", "")
        context_blocks.append(f"[{breadcrumb}]\n{text}")
    full_context = "\n\n---\n\n".join(context_blocks)
    # 5. Generation via Local LLM
    system_prompt = (
        "You are an enterprise technical assistant. "
        "Answer the user query strictly using the provided context. "
        "If the context does not contain the answer, explicitly state that you do not know.\n\n"
        f"Context:\n{full_context}"
    )
    completion = ai_client.chat.completions.create(
        model="local-model",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_query}
        ],
        temperature=0.1
    )
    return completion.choices[0].message.content
# Example Execution
if __name__ == "__main__":
    response = retrieve_and_generate(
        user_query="How do I enable TLS 1.3 in config.yaml?",
        user_department="Engineering"
    )
    print("\n🤖 Final Answer:\n", response)

The shape of a production architecture

Moving from prototype to a system that serves millions of enterprise documents means treating RAG as an engineering discipline end to end and not as a single library call. Four habits summarize it:

• Ingest cleanly. Structure beats cleverness, so use specialized parsers, keep layout breadcrumbs and skip re-indexing of unchanged files through delta hashing.
• Filter early. Security scopes and relational metadata belong in a SQL step that runs ahead of any expensive vector computation.
• Mix search strategies. Pair dense vectors with BM25 keyword matching and a cross-encoder reranker.
• Guard the boundary. Route conditionally to skip needless lookups, apply input and output guardrails, and keep humans involved in high-risk actions.