Beyond Cosine Similarity: Five RAG Retrieval Techniques Compared

Two sentences, "the cat sat on the mat" and "a feline rested on the rug", share no words at all, yet an embedding model scores them about 0.85. That's how cosine similarity earns its reputation, and it's also where most RAG tutorials stop: show cosine, paste a snippet, move on.

collection.query(query_embeddings=embedding, n_results=5)

That line works. It's also only one of five retrieval techniques, each built for a different type of problem. Relying on cosine alone means whole categories of queries fail silently, with no error, and users simply get wrong answers. Here's what cosine actually measures, where it breaks and the four other techniques, plus a way to decide which one to reach for.

What cosine measures

An embedding model converts text into a vector, a list of hundreds or thousands of numbers. Texts that mean similar things end up as vectors aimed the same way. Cosine similarity compares the angle separating two vectors rather than how far apart they are.

That's the key point. A one-line sentence and a full paragraph can mean the same thing. If their vectors point the same way, cosine reports a high score. Length, which shows up as magnitude, is noise, and direction is meaning.

Typical scores look like this:

The first row is what makes cosine impressive. No shared words, a score of 0.85. The model has learned that "cat" and "feline" live in nearby parts of semantic space, and cosine picks that up with no keyword matching at all.

Why not Euclidean distance

The obvious alternative is Euclidean distance, literally how far apart two vectors sit. The trouble: repeat a document ten times and its vector grows ten times larger, so it looks far from the original even though nothing changed in meaning. Cosine ignores that. Only direction counts.

A warning about scores

Cosine scores aren't universal. A 0.7 from all-MiniLM-L6-v2 is a different signal from a 0.7 from text-embedding-3-large, because each model has its own embedding space with its own density. Calibrate score thresholds on your own evaluation set, and don't copy numbers from a tutorial that used another model.

A festival analogy for non-engineers

Before the code, a mental model for the five techniques. You're at a music festival looking for a friend in the crowd, with three strategies available.

• By direction (cosine). You both agreed to face the stage. Same direction means same person, however far apart you stand. The angle you're pointing matters and the gap between you doesn't.
• By name tag (keyword search, BM25). You shout "anyone called Alex?" It works perfectly if the name tag matches exactly, and it misses every Alexis, Al or Sandy, who are the same person under a different label.
• By face and name tag (hybrid). You check both. Someone who matches on both rises to the top, and someone matching on one still gets noticed. It gives the broadest coverage.

The lesson most tutorials bury: no single method finds everyone. The technique you choose decides which answers, and which people, you'll permanently miss.

Five techniques, one at a time

1. Cosine similarity, the default

What it does. Compares the semantic direction of the query and document vectors.

Use it when. Queries are natural language and no exact terms are required: general RAG, question answering over documents, anything with plenty of paraphrasing.

Where it falls down. Literal strings. A product code, an invoice number or a person's name that barely appeared in training data can make cosine fail completely. The vectors for "GPT-4o" and "GPT-4" may sit close enough to produce noise rather than signal.

In code:

collection.query(
    query_embeddings=query_embedding,
    n_results=5
)

Trade-off. Meaning wins over exact wording. It's fast, reasonably accurate and needs no extra setup.

2. BM25 keyword search, the precision tool

What it does. Scores documents on how often query words occur in them, discounted by how common those words are across the corpus. It uses no embeddings, just counting.

Use it when. The query hinges on an exact identifier that has to match character for character, such as an invoice number, a model name with a release date or a CVE reference.

What it misses. Any sense of meaning. "Feline" won't match documents that say "cat". Synonyms and paraphrases are invisible to it.

A telling result. In one small from-scratch project, adding BM25 next to cosine lifted recall from 0.25 to 0.75. The reason was that real user queries contained exact product names that cosine was blurring into near-matches instead of hitting directly. Treat that as one data point, not a universal figure.

Trade-off. Exact-match precision at the price of zero semantic generalisation.

3. Hybrid search, the production default

What it does. Runs cosine and BM25 side by side and merges the two ranked lists with Reciprocal Rank Fusion (RRF). Neither method covers every query type alone.

Use it when. Queries blend natural language with specific terms, which describes almost every production system. A user asking "what's the refund policy for order #8821?" is part semantic (refund policy) and part exact match (#8821).

The fusion step. Every technique hands back its own ranked list. RRF gives every document a score based on its position in each list, 1 divided by (k plus rank), where k is a constant, commonly 60, and then adds the scores across techniques. Documents ranking well in both lists float to the top.

# Pseudocode for RRF merge
def rrf_score(rank, k=60):
    return 1 / (k + rank)

def merge_results(cosine_results, bm25_results):
    scores = {}
    for rank, doc in enumerate(cosine_results):
        scores[doc.id] = scores.get(doc.id, 0) + rrf_score(rank)
    for rank, doc in enumerate(bm25_results):
        scores[doc.id] = scores.get(doc.id, 0) + rrf_score(rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

On AWS, OpenSearch supports this natively: you set up a hybrid query with a weight parameter and it handles the fusion internally.

Trade-off. The best overall recall, at roughly double the retrieval cost, which is usually worth paying.

4. Reranking with a cross-encoder

What it does. A two-stage approach. Stage one retrieves the top 20 candidates through cosine or hybrid search. Stage two feeds each candidate together with the query into a cross-encoder, which scores relevance by seeing both at once.

Use it when. The top results are relevant but in the wrong order, and precision matters more than latency. It belongs in the second stage and never the first pass.

Why two stages. A cross-encoder judges relevance far better than cosine, but you can't run it over a whole database. Twenty model calls instead of one is acceptable. Ten thousand at query time is not.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')# Stage 1: retrieve candidates
candidates = collection.query(query_embeddings=embedding, n_results=20)# Stage 2: rerank
pairs = [(query, chunk) for chunk in candidates['documents'][0]]
scores = reranker.predict(pairs)
top_5 = sorted(zip(scores, candidates['documents'][0]), reverse=True)[:5]

A firm warning. Don't use reranking as the first pass. The latency cost is steep, since you swap one embedding lookup for twenty full model inferences.

Trade-off. The highest precision, and the highest expense, so use it as a second stage only.

5. Maximal marginal relevance (MMR) for variety

What it does. Trades off closeness to the query against sameness with chunks already picked. Each new pick must be relevant and unlike what's already selected.

Use it when. The top chunks all say the same thing, for instance a broad summary question where every result restates one fact.

The lambda knob.

• lambda = 1.0 means pure relevance, the same as standard cosine
• lambda = 0.0 means pure diversity, ignoring relevance entirely
• lambda = 0.5 is a balanced starting point to tune from

def mmr_select(query_embedding, candidates, lambda_param=0.5, k=5):
    selected = []
    remaining = list(candidates)

    while len(selected) < k and remaining:
        mmr_scores = []
        for doc in remaining:
            relevance = cosine_similarity(query_embedding, doc.embedding)
            if selected:
                redundancy = max(cosine_similarity(doc.embedding, s.embedding)
                                for s in selected)
            else:
                redundancy = 0
            mmr_scores.append(lambda_param * relevance - (1 - lambda_param) * redundancy)

        best = remaining[mmr_scores.index(max(mmr_scores))]
        selected.append(best)
        remaining.remove(best)

    return selected

Where it shines. Summaries, research questions and any case where the reader wants coverage across ideas and not depth on one.

Trade-off. It removes redundancy, but lambda needs tuning.

Choosing: let your queries decide

Before choosing a technique, study 20 real queries from your actual users or domain. Hypothetical ones don't count.

• 80% or more are plain natural language. Cosine is your default. Start there, since it's fast and accurate and needs no more infrastructure.
• 30% or more contain exact terms, codes, IDs or proper names. Add BM25. Once you see that pattern, hybrid search becomes the production default, so implement RRF fusion. The recall gain is usually quick and sizeable.
• Top results are relevant but the first one is weak. Put a cross-encoder reranker in a second stage, pulling 20 candidates and keeping the best 5, and check the added latency before you ship.
• Top results are all redundant. Add MMR at lambda 0.5, moving toward 0 for more diversity or toward 1 for more relevance. It's essential for summary and research-style questions.
• Users ask vague questions that barely describe what they want. Try HyDE (Hypothetical Document Embeddings) first. Generate a made-up answer to the question, embed that, and search with it instead of the raw query. The hypothetical answer sits closer to the ideal document than the question does.
• Building production RAG on AWS. OpenSearch with native hybrid query (BM25 plus dense vectors, RRF merge) and Cohere Rerank as a second stage is a stack many teams settle on.

What tutorials leave out

Techniques can conflict, not just add up. In one experiment, layering query expansion, which generates several variants of the query and retrieves for each, on top of hybrid retrieval seemed a sure win. More coverage should mean better results. Scores fell from 0.807 to 0.702. More retrieval signals amplify noise as readily as signal, and the extra variants pulled in adjacent but wrong chunks. Hybrid retrieval was already doing well, and expansion added interference. Add one technique at a time, measure each alone before stacking and remember that order matters, because something helpful in isolation can hurt in combination.

Dot product isn't cosine. Some vector databases default to dot product. The two match only when vectors are unit-normalised. If your embedding model doesn't normalise its output, dot product scores can't be compared across queries, so a 0.9 for one query and a 0.9 for another mean different things. Check what your database really computes. ChromaDB, for example, is generally described as using a different default metric from Qdrant, so confirm both in current documentation.

Queries decide the technique, not documents. Engineers often spend time classifying their corpus as technical or conversational, long or short. That matters for chunking. For choosing a retrieval technique, query analysis matters much more. A technical corpus searched with natural-language questions is a cosine problem, and the same corpus searched by part number is a hybrid problem.

The full pipeline

Here's the layered design many production RAG systems end up with:

User Query
    ↓
[Optional: HyDE — if query is vague]
    ↓
Stage 1 Retrieval: Hybrid (Cosine + BM25 → RRF merge)
    → Top 20 candidates
    ↓
[Optional: Cross-encoder reranking — if precision matters]
    → Top 5 candidates
    ↓
[Optional: MMR — if results are redundant]
    → Final k candidates
    ↓
LLM Context Assembly + Generation

Not every system needs every layer. Start with cosine, add hybrid when exact-match failures appear, add reranking when ordering is wrong and add MMR when results repeat themselves. Measure at every step, because what helps one system can hurt another.

The number that matters

Reports from retrieval experiments often show a recall gap between cosine-only and hybrid search on mixed query sets, with one small project seeing 0.25 climb to 0.75. If that holds even loosely for your data, it means about three times as many correct answers reach the language model before it writes a word, with no prompt engineering, no model upgrade and no infrastructure overhaul. It comes down to making the retrieval choice well.