Cutting RAG Token Spend 67% With Qdrant Multivectors and Sentence Isolation

Token bills have become a headline worry for teams running retrieval-augmented generation. Large companies are hunting for ways to cut consumption, and many teams find that an unoptimized LLM pipeline can cost more than the engineering time it was meant to save. Legal document review makes a good test bed, because the documents are long, the answers are short and the volume is high.

The page nobody needed

Picture an attorney three weeks into discovery on a contract dispute, with stacks of master services agreements and corporate filings to work through. The question is narrow: how high does the indemnification cap go when a third party claims IP infringement? A single sentence somewhere in a forty-page contract answers it.

A standard RAG system returns the top three chunks, each 500 to 1,000 tokens, and they are mostly recitals, definitions, payment terms and governing-law clauses that have nothing to do with IP claims. The one useful sentence sits in the middle. That is annoying for the reader, and at scale it is expensive: across thousands of filings in a litigation audit, the LLM provider is paid to process mountains of redundant boilerplate.

Two questions follow. How much of the retrieved text is actually necessary? And could the exact sentence be isolated without a heavy second service handling reranking?

Nothing about this waste is specific to law. Any corpus of long, templated documents, such as insurance policies, vendor contracts or compliance manuals, has the same shape: a short answer wrapped in pages of text that every retrieval pass drags into the prompt. Cost scales with the size of the retrieved chunks and not with the size of the answer, so the lever worth pulling is how much of each chunk reaches the model.

Why do the reranking inside the vector database

The usual two-stage blueprint runs a fast dense search to collect roughly the top 100 candidates, then makes a second network call to an external reranker such as Cohere, SageMaker or a custom container to sort them. It works, with three drawbacks.

• Two network calls go to two different vendors.
• If the reranker goes down, search fails with it.
• The bill covers vector storage and reranking calls.

Qdrant offers an alternative. Since version 1.10 it supports native multi-vector fields with a MAX_SIM comparator, the operator behind ColBERT-style late interaction. The token-to-token comparison runs inside the database in a single query, so the late-interaction accuracy of a separate reranker arrives without the second hop. Combined with sentence isolation, the setup cut token costs by 67%.

Memory settings complete the profile.

• Binary quantization on the dense index. Dense vectors shrink to one bit per dimension, and these small vectors stay in RAM (always_ram=True) for very fast candidate prefetching.
• ColBERT matrices on disk. The heavy token matrices live on disk (on_disk=True) and are loaded only for the top candidate set, perhaps 50 to 100 chunks, during rescoring, which keeps RAM cost low.

The pipeline

The design has six steps with one database call and no external reranking API.

1. Split the agreements by section so no chunk cuts a legal clause in half.
2. Embed every chunk twice, once as a standard dense vector for the fast pass and once as a ColBERT multi-vector matrix for token-level rescoring.
3. Prefetch the top N candidates using binary-quantized dense vectors.
4. Rescore those candidates with ColBERT MAX_SIM inside Qdrant.
5. Take the top chunks and score their individual sentences locally against the ColBERT query vector.
6. Send only the matching sentences to the LLM.

Pricing for the benchmark assumed $3.00 per million input tokens, in line with a mid-tier commercial model.

Collection schema

The configuration script below sets up standard dense embeddings and ColBERT multi-vectors side by side, with FastEmbed models loaded locally on CPU.

from qdrant_client import QdrantClient, models
# ADDED: Load FastEmbed models locally on CPU
from fastembed import TextEmbedding, LateInteractionTextEmbedding
COLLECTION_NAME = "legal_discovery"
DENSE_DIM = 384  # BAAI/bge-small-en-v1.5
COLBERT_DIM = 128  # colbert-ir/colbertv2.0
# ADDED: Instantiate the vector models
dense_model = TextEmbedding("BAAI/bge-small-en-v1.5")
colbert_model = LateInteractionTextEmbedding("colbert-ir/colbertv2.0")
client = QdrantClient("<http://localhost:6333>")
client.create_collection(
    collection_name=COLLECTION_NAME,
    vectors_config={
        "dense": models.VectorParams(
            size=DENSE_DIM,
            distance=models.Distance.COSINE,
            quantization_config=models.BinaryQuantization(
                binary=models.BinaryQuantizationConfig(always_ram=True),
            ),
        ),
        "colbert": models.VectorParams(
            size=COLBERT_DIM,
            distance=models.Distance.COSINE,
            multivector_config=models.MultiVectorConfig(
                comparator=models.MultiVectorComparator.MAX_SIM
            ),
            on_disk=True,
            hnsw_config=models.HnswConfigDiff(m=0),
        ),
    },
)

One setting stands out in the ColBERT vector parameters: an HNSW config with m set to 0. Since ColBERT is used only to rescore a pre-filtered pool, no HNSW graph is needed for it. Skipping the graph saves a lot of index-building time and storage.

One round trip for prefetch and rescore

Qdrant's universal Query API lets the first-stage prefetch and the second-stage rescore run as a single operation. No query, ID parsing and second hop to a reranker are needed; the whole flow is orchestrated in one database round trip.

# ADDED: Generate query embeddings (ColBERT uses query_embed to add prefix padding)
dense_query = next(dense_model.query_embed(query)).tolist()
colbert_query = next(colbert_model.query_embed(query)).tolist()
# Run the two-stage query in one network round-trip
results = client.query_points(
    collection_name=COLLECTION_NAME,
    prefetch=models.Prefetch(
        query=dense_query,
        using="dense",
        limit=prefetch_limit,
        params=models.SearchParams(
            quantization=models.QuantizationSearchParams(rescore=False),
        ),
    ),
    query=colbert_query,
    using="colbert",
    limit=top_k,
    with_payload=True,
)

The rescore=False flag inside the prefetch block deserves attention. With binary quantization, Qdrant would normally fetch candidates with the quantized vectors and then load the full-precision float32 dense vectors to rescore them. Because the ColBERT stage does a more exact job anyway, turning dense rescoring off avoids reading full-precision vectors from disk and trims database overhead.

From chunk to sentence

MAX_SIM gives one aggregate score per chunk and does not reveal which tokens drove it. To pull out only relevant text, a final local step splits the winning chunks into sentences and computes ColBERT late-interaction scores for each. Since this runs on only the final few chunks, where the text is small, no database scaling bottleneck appears, and sentence-level precision comes cheaply on the CPU.

import re
import numpy as np
# ADDED: Basic sentence splitter regex
SENTENCE_SPLIT = re.compile(r"(?<=[.;])\s+(?=[A-Z])")
def max_sim(query_vecs: np.ndarray, doc_vecs: np.ndarray) -> float:
    # Compute token-to-token similarity matrix
    sims = query_vecs @ doc_vecs.T  # (num_query_tokens, num_doc_tokens)
    # Sum the maximum similarity scores along the document axis
    return float(sims.max(axis=1).sum())
def isolate_sentences(chunk_text: str, query_vecs: np.ndarray, colbert_model, top_n: int = 1):
    # ADDED: Split chunk text into candidate sentences
    sentences = [s.strip() for s in SENTENCE_SPLIT.split(chunk_text) if len(s.strip()) > 15]
    if not sentences:
        return [(chunk_text, 0.0)]
    # Embed each sentence locally using ColBERT
    sentence_vecs = list(colbert_model.embed(sentences))
    scored = [(sentences[i], max_sim(query_vecs, sentence_vecs[i])) for i in range(len(sentences))]
    scored.sort(key=lambda pair: pair[1], reverse=True)
    return scored[:top_n]

The local calculation is fast because it only touches the top three or five chunks. Once the best sentences are identified, the rest of each chunk is dropped and the prompt payload is assembled.

def build_optimized_prompt(query: str, chunk_texts: list[str], colbert_model) -> str:
    query_vecs = next(colbert_model.query_embed(query))
    context_parts = []
for i, text in enumerate(chunk_texts):
        top_sentences = isolate_sentences(text, query_vecs, colbert_model, top_n=1)
        isolated_text = " ".join(s for s, _ in top_sentences)
        context_parts.append(f"[Source Chunk {i+1}]: {isolated_text}")
    context_str = "\n\n".join(context_parts)
    return f"Context:\n{context_str}\n\nQuestion: {query}\nAnswer:"

Chunk size and the honest number

Token reduction alone is a tempting metric. Packing ten to twelve legal sections into one 2,000-word chunk and then applying sentence isolation can theoretically claim a reduction above 85%. In a production legal system that approach hurts retrieval accuracy. A big chunk drowns the signal of a specific clause, such as a third-party IP indemnification cap, in surrounding boilerplate within the dense vector space. Queries against these muddy, multi-clause chunks tend to return unrelated sections like warranties, because the dense retriever cannot isolate the target.

The fix is a strict boundary of one legal clause per chunk, with every chunk holding exactly one distinct section and realistic clause lengths. Under that scheme the benchmark gave a verified 67.1% reduction in prompt tokens. Manual checks on indemnification and IP-ownership queries confirmed the right clauses were found, even though the headline figure fell from a theoretical 87.3% to a real-world 67.1%. Clean section-level boundaries matter more than a larger percentage.

The latency trade

Binary quantization suggests the two-stage pipeline might beat plain dense search on speed. It does not. Across repeated runs the two-stage search was consistently two to four times slower, depending on the machine and cache warmth: 39.5 ms against 20.4 ms. The reason is straightforward. Dense-only search is a single embedding lookup, while the two-stage version runs both query encoders, dense and ColBERT, and performs a token comparison matrix inside the database. A separate reference run of dense-only search averaged 7.4 ms.

A latency sweep that raised the prefetch limit from 10 candidates to 500 barely changed anything, and latency stayed in the same band. The two-stage structure itself, and not the size of the prefetch window, drives the extra time.

The bigger picture is that end-to-end RAG latency is dominated by LLM generation, not by database queries. Saving more than 400 tokens per query shortens downstream processing and generation by hundreds of milliseconds, so the whole application ends up noticeably faster.

What the savings are worth

At $3.00 per million input tokens, a 67.1% cut in prompt tokens adds up. For a litigation-discovery run of 500,000 queries, that is more than $600 saved per case audit. Shorter prompts also lessen the lost-in-the-middle effect, where a model overlooks relevant text buried in a long context.

What to check before adopting it

Three measurements decide whether the design fits a given workload. The first is answer correctness on a hand-checked query set, because token savings mean little if the right clause is missing. The second is the latency of the whole request including generation, since the extra database time is small next to the model call. The third is the token count per prompt before and after sentence isolation, measured at the pricing in use. Results from one document type may not carry over to another, so the benchmark should be repeated on the target corpus.

Lessons for production

• Keep reranking inside the database. External reranking APIs add points of failure and network round trips, while a native MAX_SIM operator handles the work in one trip.
• Switch off HNSW for rescore-only fields. Setting m=0 on multi-vector configs avoids an expensive indexing step that brings no retrieval benefit for ColBERT matrices.
• Distrust very large chunks. Aggressive sizes can show impressive token reduction on paper while diluting semantic representation and damaging accuracy.
• Use a real sentence tokenizer. A raw regex is fine for a demo, but production code should use a library like NLTK or spaCy that copes with legal abbreviations such as Inc., Sec. and Exhibit A. without splitting sentences wrongly.

Native multi-vector comparison plus local sentence-level scoring can produce a precise, cost-conscious pipeline that returns the clause a user needs without inflating the monthly API bill.