Semantic Caching for LLM Agents With Redis: Cut Cost and Latency

Calls to a strong LLM cost real money, and they're slow. Anyone who has watched an API bill grow after putting an agent in front of real users knows the tension: the models that answer best are the ones that are most expensive and slowest. GPT-4-class models can run to several dollars per thousand requests, and agentic workflows make it worse, because one user query may trigger anywhere from 8 to 20 model calls.

Semantic caching goes after that cost directly, and most teams underuse it. It's easy to build with Redis.

Why ordinary caches miss

A regular cache treats two requests as the same only if their bytes match. That works for database queries and HTTP requests and falls apart for natural language. Imagine a support agent that receives these from three different users:

• A customer asks how to request a refund
• Another demands their money back
• A third wants to know what the refund policy says

A byte-level cache sees three unrelated inputs and pays for three LLM calls. A person would say all three want the same answer. A semantic cache works from meaning instead of characters.

The mechanism in four steps

1. Embed the incoming query, turning it into a high-dimensional vector with an embedding model.
2. Compare that vector against every stored entry, using cosine distance or a similar measure.
3. If the nearest entry falls inside a distance threshold you've configured, count it as a hit and return the stored answer straight away.
4. If nothing is close enough, send the query through your RAG pipeline or the LLM, then store the question and answer for next time.

Paraphrased questions end up sharing answers. As the cache fills, a shrinking share of traffic reaches the model.

A minimal version from scratch

Before using a production library, it helps to see the moving parts. Start with the imports:

from sentence_transformers import SentenceTransformer
import numpy as np

Then embed a small FAQ set and write a nearest-neighbor lookup:

model = SentenceTransformer("all-mpnet-base-v2")# Embed your FAQ dataset
faq_questions = ["How do I get a refund?", "Where is my order?", ...]
faq_embeddings = model.encode(faq_questions)def cosine_distance(a, b):
    return 1 - np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))def check_cache(query: str, threshold: float = 0.3):
    query_embedding = model.encode(query)
    distances = [cosine_distance(query_embedding, e) for e in faq_embeddings]
    best_idx = np.argmin(distances)
    best_distance = distances[best_idx]

    if best_distance < threshold:
        return faq_answers[best_idx]  # Cache hit
    return None  # Cache miss

Try a test question about how many days a refund takes to arrive and the lookup returns the stored FAQ about getting a refund as its closest entry, at a cosine distance of roughly 0.33. Nothing was matched on shared words alone; the vectors simply sit near each other.

Running it on Redis

The toy version has no persistence, no expiry, no separation between tenants and no optimized vector search. For anything customer-facing you want a real database. Redis, together with the open-source RedisVL SDK, supplies those pieces out of the box.

from redisvl.extensions.cache.llm import SemanticCache
from redisvl.utils.vectorize import HFTextVectorizer

Pick an embedding model that was built for the job, then create the cache:

# Load a cache-optimized embedding model
vectorizer = HFTextVectorizer("redis/langcache-embed-v1")# Create the cache
cache = SemanticCache(
    name="customer_support_cache",
    vectorizer=vectorizer,
    redis_client=redis_client,
    distance_threshold=0.3
)# Set TTL (time to live) — keeps cache fresh
cache.set_ttl(86400)  # 24 hours

The langcache-embed-v1 model is an open-weight embedding model fine-tuned for semantic caching, so it tends to beat general-purpose embedders on this task.

Three production features come with the Redis route:

• TTL. Entries expire on their own, so stale answers don't linger as the product changes.
• Namespacing. Teams, users or tenants can each have an isolated cache within one Redis instance.
• Optimized vector search. HNSW indexing gives low-latency nearest-neighbor lookups even at scale.

Four numbers to watch

Building the cache is the easy half. You also need to know it's working correctly.

Hit rate is the share of queries that find a match within the threshold, and it sets your savings. At a 30% hit rate on a system making 10,000 LLM calls a day, 3,000 calls disappear.

Precision asks what fraction of hits returned the right answer. A cache that confidently serves wrong answers is worse than none.

Recall asks what fraction of the queries that should have hit actually did, given that an equivalent entry existed. Low recall leaves savings unclaimed.

Latency improvement compares the two paths. Benchmarks put cache responses at roughly 2 to 5 ms against 300 to 500 ms for a typical GPT-4 call, about 160 times faster. What you actually feel depends on how often you hit, as this formula shows:

With-Cache Latency = (avg_llm_latency × (1 - hit_rate)) + (avg_cache_latency × hit_rate)

At 30% hits, expect overall latency to drop by something like 25 to 30%. Once the cache warms up to 60 to 75%, the gains get dramatic.

Treating the cache as a classifier

Every lookup ends in one of four outcomes:

False positives are the dangerous ones: two questions that look alike but mean different things, so the cache hands back a wrong answer. A question about refunds after a change of heart and a question about the general refund process may land close together in embedding space, yet each deserves its own reply. That's the case where the cache has to be most careful, because the answer it returns sounds completely plausible.

Raising accuracy

Sweep the threshold. Distance threshold is the most influential setting. A lower value raises precision and lowers recall, while a higher one does the opposite. Try a range of values on a representative test set and pick the one that maximizes F1, the harmonic mean of precision and recall. That approach lifted precision from 79% to 90% while holding a 25% hit rate.

Rerank with a cross-encoder. Standard embedding models encode each sentence on its own. A cross-encoder reads the query and a candidate together, which catches subtleties that bi-encoders miss. Take one sentence about a bank raising interest rates and another about a river flooding near the bank after heavy rain: the shared word "bank" pulls their embeddings fairly close, but a cross-encoder sees they're unrelated. In experiments reranking pushed precision to 94%. Cross-encoders can't produce searchable vectors, so they work as a second stage over the top-K candidates that the embedding model retrieves.

Ask a small LLM to judge. For maximum precision, have a cheap model such as GPT-4o-mini validate borderline hits with a plain prompt: are these two questions semantically equivalent? It answers in milliseconds for a fraction of a full call's price. Testing showed 100% precision, with lower recall since the judge sometimes rejects valid matches. Whether that's acceptable depends on the use case. In customer support, a slightly wrong cached answer can be worse than a fresh call.

Fuzzy-match first. Before computing any embedding, run edit-distance matching of the query against stored entries. If it differs from one by two or three transposed characters, it's almost certainly a typo of the same question, so match it directly. That skips the embedding step entirely, taking latency near zero. At high similarity thresholds this reached 100% precision and 94% recall on test sets full of typos.

The bigger win: caching inside agents

Caching only the final response to a user question is an incomplete mental model. Agents work differently. A typical flow for a complicated support question looks like this:

1. Break the query into 3 to 5 sub-questions.
2. Retrieve relevant documents from a knowledge base.
3. Ask an LLM to synthesize an answer.
4. Judge the answer's quality and iterate if needed.

Every sub-question is a chance to cache. If the agent has already answered "What are your API rate limits?", it needn't ask the model again when that sub-question turns up inside a different user's complex request.

A LangGraph-based deep research agent was tested in three scenarios:

By the third scenario, three of every four sub-questions came from cache. Model calls fell from 8 to between 4 and 6, and end-to-end latency dropped by around a third. Response quality stayed the same, with personalized and accurate answers either way.

The effect compounds. Each successful run adds to the cache, and later users with similar questions benefit from everything before them, so savings grow as the system handles hundreds or thousands of queries.