Semantic caching with Redis for vector search

Building an AI product often surprises teams less with model quality than with the invoice. A chatbot that looks cheap at a few hundred users can suddenly spend heavily once every search, recommendation, and question triggers another embedding call.

Many of those calls are redundant. Semantic caching reuses earlier results when a new query means nearly the same thing. On Redis, that pattern can cut latency and infrastructure spend without rewriting application business logic.

The hidden cost of vector search

A typical retrieval-augmented generation (RAG) path looks like this:

User Query
     ↓
Embedding API
     ↓
Vector Database
     ↓
Retrieved Documents
     ↓
LLM

Each fresh query usually means:

• calling an embedding model
• paying network latency
• running a vector search
• returning hits

Embedding calls cost money. Round trips cost time.

Consider two searches minutes apart:

• “best budget gaming laptop”
• “cheap gaming laptops”

Wording differs; intent barely does. Yet many stacks recompute both from scratch. Across thousands of users, the platform pays repeatedly for almost the same work.

What is semantic caching?

Classic caches key on exact strings. Cache “red shoes,” and only the identical phrase hits. “shoes that are red” misses.

Semantic caching compares meaning. Each query becomes an embedding. When two vectors are close enough, the stored answer can be reused.

For example:

Query A
"best beginner Python book"
↓
Embedding A

Later:

Query B
"good Python books for beginners"

If cosine similarity clears a chosen threshold—say 0.95—the system returns the cached payload. No new embedding call, no fresh vector search, no extra bill line.

Building a two-level cache

Production designs often stack two caches.

Level 1: embedding cache

The fastest tier. An identical query string returns a stored embedding instead of calling the model again.

Query
   ↓
Redis
   ↓
Embedding

That alone removes one of the most common paid calls in AI apps.

Level 2: semantic result cache

The second tier stores final search answers. Matching is by embedding neighborhood, not raw text.

Embedding
      ↓
Redis Vector Search
      ↓
Cached Search Results

A close neighbor means the vector database is skipped and the user gets a near-instant cached answer.

Why Redis?

Redis is a fast in-memory store. With Redis Stack and RediSearch, it can also run vector similarity search.

That combination lets Redis:

• store embeddings
• compare vectors
• return nearest neighbors
• expire entries with TTL
• serve hits in a few milliseconds

The cache stops being a blunt key-value bucket and becomes a semantic lookup layer.

Choosing the right similarity threshold

Threshold choice dominates quality. At 0.99, only near-duplicates match—high precision, fewer hits. At 0.85, hit rate rises, but unrelated queries can collide (running shoes versus basketball shoes).

Tolerance depends on domain. E-commerce can often accept looser thresholds; medical search usually cannot. Some systems even set different thresholds per collection.

Keeping the cache fresh

Cached answers help only while they stay true. “Latest iPhone prices” can go stale overnight without a refresh plan. Two common approaches cover most cases.

Time-based expiration

Assign a TTL so Redis deletes entries automatically—good for frequently changing catalogs.

Event-based invalidation

When the application knows a change happened—inventory updates, price edits, new documents, removals—it deletes affected keys immediately instead of waiting for TTL.

How Redis searches embeddings

Modern Redis stores embeddings as float vectors and indexes them with structures such as:

• FLAT
• HNSW

A new query does not brute-force every vector; indexes locate neighbors quickly, so semantic lookups often finish in a few milliseconds.

Putting everything together

An end-to-end request path looks like:

User Query
      ↓
Embedding Cache
      ↓
Hit?
      ↓
Yes → Use Cached Embedding
No
 ↓
Embedding API
 ↓
Redis Semantic Cache
 ↓
Hit?
 ↓
Yes → Return Cached Results
No
 ↓
Vector Database
 ↓
Store Results in Redis
 ↓
Return Results

The expensive embedding-plus-search path runs only on misses. Everything else stays short.

What kind of improvement can you expect?

Numbers vary by product, but many production reports look like:

• general semantic search often lands around a one-fifth to one-third hit rate
• autocomplete and repeated queries can climb toward roughly half of traffic
• cache hits commonly answer in single-digit milliseconds
• a miss that embeds and searches end-to-end may take a few hundred milliseconds

Each hit removes an embedding API call. At thousands of requests, that compounds into real spend and latency savings.

Common pitfalls

Cache poisoning

A bad answer that lands in cache can poison similar future queries. Monitoring plus sensible expiration reduce the blast radius.

Memory growth

Embeddings are large: hundreds or thousands of floats each. Collections grow fast. Set memory ceilings and eviction policies before go-live.

Cold starts

A fresh deploy starts empty, so early traffic pays full price. Popular queries warm the cache over time; some teams preload common searches at deploy to shorten that ramp.

When semantic caching isn’t the right choice

The pattern shines when many users ask similar questions. It weakens when answers are strongly personal or ultra-fresh:

• “Restaurants nearby”
• personalized recommendations
• user-specific dashboards
• live inventory
• rapidly changing prices

Serving a neighbor’s cached semantic hit can be simply wrong in those cases.

Final thoughts

As AI traffic grows, infrastructure cost often competes with model quality for attention. Semantic caching is a pragmatic cut: recognize shared intent, skip repeated embedding and search work. Redis makes that practical by pairing in-memory speed with native vector search—faster responses, fewer paid embedding calls, and cleaner scaling. Sometimes the best optimization is not a better model, but skipping work that never needed to run.