Practical notes: How Qdrant Reduced RAG Token Costs by 67% with Native ColBERT

The following notes reconstruct a practical path around “How Qdrant Reduced RAG Token Costs by 67% with Native ColBERT Reranking”. Emphasis stays on contracts, checks, and drop-in code placeholders rather than motivational framing. When working through the Overview stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

The Page We Don’t Need

The The Page We Don stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices.

Why Qdrant Over Everything Else?

The Why Qdrant Over Everything stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices.

What We Are Actually Building

The What We Are Actually stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices. The What We Are Actually stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

Cost & Performance Benchmark Summary

For the Cost Performance Benchmark Summary stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.

Setting Up the Collection Schema

For the Setting Up the Collection stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.

from qdrant_client import QdrantClient, models
# ADDED: Load FastEmbed models locally on CPU
from fastembed import TextEmbedding, LateInteractionTextEmbedding
COLLECTION_NAME = "legal_discovery"
DENSE_DIM = 384  # BAAI/bge-small-en-v1.5
COLBERT_DIM = 128  # colbert-ir/colbertv2.0
# ADDED: Instantiate the vector models
dense_model = TextEmbedding("BAAI/bge-small-en-v1.5")
colbert_model = LateInteractionTextEmbedding("colbert-ir/colbertv2.0")
client = QdrantClient("<http://localhost:6333>")
client.create_collection(
    collection_name=COLLECTION_NAME,
    vectors_config={
        "dense": models.VectorParams(
            size=DENSE_DIM,
            distance=models.Distance.COSINE,
            quantization_config=models.BinaryQuantization(
                binary=models.BinaryQuantizationConfig(always_ram=True),
            ),
        ),
        "colbert": models.VectorParams(
            size=COLBERT_DIM,
            distance=models.Distance.COSINE,
            multivector_config=models.MultiVectorConfig(
                comparator=models.MultiVectorComparator.MAX_SIM
            ),
            on_disk=True,
            hnsw_config=models.HnswConfigDiff(m=0),
        ),
    },
)

The Unified Query Pipeline

For the The Unified Query Pipeline stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call. For the The Unified Query Pipeline stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

# ADDED: Generate query embeddings (ColBERT uses query_embed to add prefix padding)
dense_query = next(dense_model.query_embed(query)).tolist()
colbert_query = next(colbert_model.query_embed(query)).tolist()
# Run the two-stage query in one network round-trip
results = client.query_points(
    collection_name=COLLECTION_NAME,
    prefetch=models.Prefetch(
        query=dense_query,
        using="dense",
        limit=prefetch_limit,
        params=models.SearchParams(
            quantization=models.QuantizationSearchParams(rescore=False),
        ),
    ),
    query=colbert_query,
    using="colbert",
    limit=top_k,
    with_payload=True,
)

Getting from Chunk to Sentence

When working through the Getting from Chunk to stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.

import re
import numpy as np
# ADDED: Basic sentence splitter regex
SENTENCE_SPLIT = re.compile(r"(?<=[.;])\s+(?=[A-Z])")
def max_sim(query_vecs: np.ndarray, doc_vecs: np.ndarray) -> float:
    # Compute token-to-token similarity matrix
    sims = query_vecs @ doc_vecs.T  # (num_query_tokens, num_doc_tokens)
    # Sum the maximum similarity scores along the document axis
    return float(sims.max(axis=1).sum())
def isolate_sentences(chunk_text: str, query_vecs: np.ndarray, colbert_model, top_n: int = 1):
    # ADDED: Split chunk text into candidate sentences
    sentences = [s.strip() for s in SENTENCE_SPLIT.split(chunk_text) if len(s.strip()) > 15]
    if not sentences:
        return [(chunk_text, 0.0)]
    # Embed each sentence locally using ColBERT
    sentence_vecs = list(colbert_model.embed(sentences))
    scored = [(sentences[i], max_sim(query_vecs, sentence_vecs[i])) for i in range(len(sentences))]
    scored.sort(key=lambda pair: pair[1], reverse=True)
    return scored[:top_n]

def build_optimized_prompt(query: str, chunk_texts: list[str], colbert_model) -> str:
    query_vecs = next(colbert_model.query_embed(query))
    context_parts = []
for i, text in enumerate(chunk_texts):
        top_sentences = isolate_sentences(text, query_vecs, colbert_model, top_n=1)
        isolated_text = " ".join(s for s, _ in top_sentences)
        context_parts.append(f"[Source Chunk {i+1}]: {isolated_text}")
    context_str = "\n\n".join(context_parts)
    return f"Context:\n{context_str}\n\nQuestion: {query}\nAnswer:"

The Golden Rule of Chunk Size: Why Chunk Boundaries Matter for Accuracy

When working through the The Golden Rule of stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.

The Tradeoff Speaks for Itself

When working through the The Tradeoff Speaks for stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn. When working through the The Tradeoff Speaks for stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

What’s the Financial Impact

The What s the Financial stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices.

Design Takeaways for Production

The Design Takeaways for Production stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices.

GitHub

The GitHub stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices. The GitHub stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.

References

For the References stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.

Operational checklist

For the Operational checklist stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state.

Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.

Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.

Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.

Pin dependency versions and record the image digest that ran the demo. Reproducibility beats tribal knowledge.

Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.

Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.

Batch note for 98b4b4d4d553: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.