Semantic Caching for LLM Apps on Ubuntu 24.04 With Qdrant

Once a generative AI app goes live, the bill gets your attention fast, because every token is charged. Read through your prompt logs and you'll likely find that up to 40% of user queries carry the same intent in different words. "Steps to reset a password?" and "Locked out of the account, need access back" mean the same thing. A classic key-value store sees two unrelated requests, calls the OpenAI API twice and bills you in full both times.

A semantic cache fixes that. Here's how to build one on Ubuntu 24.04 LTS, from an operations point of view, along with the three traps that tutorials usually skip.

Exact matching versus meaning

Two lookup styles behave very differently:

• Lexical caching, such as an ordinary Redis key-value store, needs identical strings. "Where is the order?" gets cached, but "Track the package" misses, because the characters differ.
• Semantic caching turns prompts into high-dimensional vectors with an embedding model. "Where is the order?" and "Track the package" land close together in vector space, so the stored answer comes back right away. Lookup latency drops from around 3000 ms to about 15 ms.

Set up the vector cache with Qdrant

For the store, run Qdrant in Docker with a persistent volume, then prepare a Python environment:

# 1. Install Qdrant via Docker on Ubuntu 24.04 (Persistent Storage)
sudo apt update && sudo apt install docker.io -y
sudo docker run -d --restart unless-stopped -p 6333:6333 -p 6334:6334 \
    -v $(pwd)/qdrant_storage:/qdrant/storage:z \
    qdrant/qdrant

# 2. Setup Python environment
python3 -m venv llm_cache_env
source llm_cache_env/bin/activate
pip install qdrant-client sentence-transformers openai

Keep tenants apart

A single global semantic cache inside a multi-tenant SaaS product invites serious leaks between customers. Picture User A asking for an overview of the latest transactions on their account. The model writes a reply containing A's private financial details, and it's cached. Later User B asks for a summary of their own transactions. The cache sees a 98% intent match and hands B the reply written for A.

The remedy is to put a strict namespace, tenant_id, into the Qdrant payload and run every similarity search beneath a hard metadata filter. Without that filter, a lookup should never execute. The code below creates the collection with the right vector size (384 for the local MiniLM encoder) and performs a tenant-filtered search:

from qdrant_client import QdrantClient
from qdrant_client.http import models
from sentence_transformers import SentenceTransformer

# Initialize Local Embedding Model (Free & Fast)
encoder = SentenceTransformer("all-MiniLM-L6-v2")
client = QdrantClient(host="localhost", port=6333)
COLLECTION_NAME = "semantic_cache_prod"
# Explicitly create collection with correct Vector Dimensions (384)
if not client.collection_exists(collection_name=COLLECTION_NAME):
    client.create_collection(
        collection_name=COLLECTION_NAME,
        vectors_config=models.VectorParams(
            size=384,  # Matches 'all-MiniLM-L6-v2' output dimension
            distance=models.Distance.COSINE
        ),
    )
def check_secure_cache(user_query: str, tenant_id: str, threshold: float = 0.90):
    query_vector = encoder.encode(user_query).tolist()

    # SRE Fix: Hard Metadata Filtering by tenant_id
    hits = client.search(
        collection_name=COLLECTION_NAME,
        query_vector=query_vector,
        query_filter=models.Filter(
            must=[models.FieldCondition(
                key="tenant_id",
                match=models.MatchValue(value=tenant_id)
            )]
        ),
        limit=1,
        score_threshold=threshold
    )

    if hits:
        print(f" Secure Cache HIT! (Score: {hits[0].score:.3f})")
        return hits[0].payload["llm_response"]

    return None

The threshold trap

Picking the similarity cutoff is a balance between precision and recall. Many guides recommend one blanket value of 0.80, and that produces nasty wrong answers. Closeness in embedding space doesn't mean equal meaning. "Name the capital of France" and "Name the capital of Germany" share a sentence structure, so they sit very near each other, with cosine similarity above 0.85. At a cutoff of 0.80, someone asking about Germany would receive the cached answer about France.

Tune the threshold per route instead:

• 0.95 and up for strict factual, technical or financial questions
• 0.88 for generic conversational FAQs

When disk speed becomes the bottleneck

Vector databases built on HNSW indexes lean hard on memory and disk throughput. Running heavy similarity search on a public cloud typically means block storage with provisioned IOPS, and those io2 charges can be steep.

One way around it is to host the whole cache stack on dedicated bare-metal servers. Direct-attached enterprise NVMe drives on PCIe deliver millions of raw IOPS at no extra charge, and a large pool of DDR5 RAM on dedicated hardware can keep embedding lookups under 15 ms even with heavy concurrent traffic. That removes the storage tax cloud providers add on top of your compute.