When RAG Retrieves the Wrong Document at 0.89 Similarity: Proximity vs Relevance

A familiar scene in retrieval-augmented generation: chunks are embedded and indexed, a user asks a question, the query is embedded, cosine similarity pulls the top-k chunks, they go into the prompt, and the LLM answers. The best chunk scored 0.89, which feels confident, and the answer is still wrong.

Anyone who has shipped more than one RAG system has seen this. It recurs often enough that no amount of chunking tweaks or embedding swaps removes it. So the useful question is whether the system measures relevance or only geometric proximity.

An August 2026 review of RAG technology in the journal Symmetry names this tension as one of four structural contradictions in the field and calls it the geometric-semantic mismatch. It helps to be exact about what that label claims. It does not say vector search is broken, and it is not a proven theorem about what geometry can do. It is a way of naming a limitation that turns up empirically across very different retrieval setups.

The pipeline in brief

Remove the tooling and RAG is two stages joined together.

• At indexing time, documents are chunked, each chunk is run through an encoder into a dense vector, and the vectors go into an index such as FAISS or HNSW.
• At query time, the question is embedded with the same encoder, similarity to every chunk vector is computed, the top-k are taken and handed to the LLM as context.

One load-bearing assumption holds the whole thing up: distance in vector space tracks semantic relevance in language. Texts with similar meaning should land close together and texts with different meanings far apart.

Most of the time that holds well enough to be useful, which is why dense retrieval works at all. Trouble appears at the edges, and in production the edges come up more than expected. Whenever a retriever ranks something highly, it is answering whether the passage is close in vector space. It is not answering whether the passage resolves the question. Usually both answers agree, and the interesting cases are the ones where they do not.

Similarity is not relevance

Cosine similarity measures how closely two vectors point the same way. It does not directly measure whether a passage answers the question. The two overlap enough for dense retrieval to be genuinely useful, yet they are different things, and a meaningful share of retrieval failures sits in the gap.

The observation is old. The 2020 paper that introduced DPR gives a telling pair of numbers. Its dense dual-encoder retriever scored 78.4 percent top-20 accuracy on Natural Questions, then fell to 63.2 percent on SQuAD, where plain BM25 managed 68.8 percent. The authors blamed training-distribution bias: SQuAD questions share unusually many words with their answer passages, a pattern that a lexical matcher exploits better than a dense encoder trained on broader data.

One data point does not establish a universal law. It does show concretely that semantic similarity and lexical precision can pull apart on the same benchmark with the same retriever. The Symmetry review generalizes the pattern: a mechanism tuned for similar meaning is not automatically tuned for the one word that matters, and the reverse also holds. Whether that stretches across every retrieval setting is best treated as a working hypothesis. It does fit a lot of practitioner experience.

Where the geometry bends

A level deeper lies the geometric issue itself. Dense retrieval maps text into a continuous vector space, nearly always Euclidean because dot-product and cosine similarity assume it. Euclidean space is good at saying two things are alike along some dimensions. It copes less well with at least three patterns that appear constantly in real text.

• Hierarchy. A dog is a mammal, which is an animal. A flat vector space has no native idea of is-a containment, and approximating a tree with straight-line distances distorts more as the hierarchy deepens.
• Synonymy versus near-synonymy. Cheap and inexpensive should sit close. Cheap and poorly made arguably should not, yet they co-occur so often in training text that their vectors can be drawn nearer than meaning alone justifies.
• Time. The CEO of a company is a different person depending on when the sentence was written. A static embedding has no axis for a fact that expires, unless something upstream encodes it.

One remedy proposed in the literature is hyperbolic space instead of Euclidean. Distances there grow exponentially outward, so tree-like hierarchies are represented with less distortion, mapping naturally onto many levels removed. Hyperbolic dense retrieval methods have reported better generalization on conceptual-hierarchy tasks. It is a trade and not a cure, however. Compute costs rise, tooling is much less mature, and hierarchy is only one of the three patterns, so a corpus dominated by synonymy or temporal drift may gain nothing from changing geometry.

That is what the mismatch really points to: different geometric assumptions trade off which semantic relationships get represented well. Choose Euclidean and hierarchy tends to suffer. Lean hyperbolic and the price is compute and immature tooling, while synonymy and time remain unaddressed. Whatever space is chosen, the choice implicitly decides which relationships the retriever handles well and which it is likelier to misjudge.

An example with documentation

Imagine a product documentation set and the question: how do I reset my API key?

The retriever returns a chunk titled how to reset an account password at 0.87 cosine similarity. That is high, because reset, API key and password all live in a tight neighborhood of authentication and account management. The chunk on API key rotation sits in another section, is phrased with less generic vocabulary, and scores 0.81.

The wrong document came back first, with a score that would clear any naive relevance threshold. The embedding space correctly noticed that both concern account security. It had no way to tell the specific procedure for one thing from the specific procedure for another when the two are siblings in one semantic cluster.

The same pattern scales to contract clauses that differ by a single negation, dosage instructions that differ by patient subgroup, and financial terms that differ by fiscal year. Each time, closeness in the vectors reflects a shared subject and says nothing about whether the answer matches.

Retrieval ranks candidates and does not deliver a verdict

Many RAG pipelines quietly assume the top-1 or top-k result is the answer to where the information lives. It would be better to see it as a candidate that still has to earn a place in the prompt.

A retriever that returns the password page for a question about an API key has not necessarily failed at its own job. Strictly speaking, its task is an inexpensive first cut: shrink an enormous collection to a handful of plausible candidates. The weak point is a design that accepts the first result as ground truth and examines it no further. The chain does not have to be query, retriever, top-1, LLM. It can as easily run query, then a cheap high-recall retriever, then a candidate set, then a slower and more precise reranker, then the best candidate, then the LLM.

Retrieval failure and generation failure are related but separate, and mixing them up makes the mismatch seem worse than it is. A single-shot top-1 pipeline cannot recover once the ranking is off. A pipeline that treats retrieval as candidate production and hands the real relevance judgment to a second stage has a chance of catching the near-miss, a document close enough in vector space to look like a winner and wrong enough in substance to matter.

Why production notices

This is no merely academic framing. The gap between proximity and relevance produces concrete and expensive failures.

• Silent amplification of hallucination. The LLM cannot tell that a chunk is adjacent to the topic but wrong. It writes a fluent, confident answer from the wrong source, and confident and wrong is worse than saying it does not know for nearly any downstream use.
• Falling precision on hierarchical corpora. Both anecdote and literature suggest that the deeper and more structured a knowledge base is, with nested policies, versioned specs or org charts, the more flat vector search struggles. The review proposes a formal diagnostic, measuring how precision degrades as hierarchy depth grows. It is offered as a useful metric and not a settled curve, and it is better measured on the corpus at hand than assumed.
• Invisible in the eval set until it costs money. Standard benchmarks such as recall@k on generic question pairs do not stress hierarchical or near-synonym distinctions the way a real domain corpus does. The internal suite passes, then a version of the password-versus-API-key problem appears in the third week of production.

Mitigations

None of this means dense retrieval is broken. It means retrieval by a single mechanism is under-specified for real corpora. These mitigations are listed roughly by how much architecture each needs.

Hybrid retrieval, lexical plus dense. Run BM25 and dense retrieval in parallel and fuse the scores, for instance with reciprocal rank fusion, a common and simple choice. That patches the lexical-overlap blind spot directly and leaves the embedding model alone.

# Minimal hybrid retrieval sketch: dense + BM25 fused via RRF
from rank_bm25 import BM25Okapi
import numpy as np

def reciprocal_rank_fusion(dense_ranked_ids, bm25_ranked_ids, k=60):
    scores = {}
    for rank, doc_id in enumerate(dense_ranked_ids):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(bm25_ranked_ids):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: -x[1])# dense_ranked_ids  = [doc_id, ...] from your vector store's top-k search
# bm25_ranked_ids   = [doc_id, ...] from bm25.get_top_n(query_tokens, doc_ids, n=k)
fused = reciprocal_rank_fusion(dense_ranked_ids, bm25_ranked_ids)

Cross-encoder reranking. Use dense retrieval for cheap, high-recall candidate generation of 50 to 100 items, then rerank with a cross-encoder that attends to query and document together. It costs more per pair but is much better at the topically close, factually wrong case, since it does not depend on a single fixed vector for each text.

Query rewriting or decomposition. RQ-RAG is one example: the system is trained to restate vague or multi-part questions before any search happens. They help when the mismatch comes from an underspecified question and not from awkward geometry in the corpus.

Structure-aware retrieval in the GraphRAG style. For genuinely hierarchical or relational material such as org charts, clause dependencies or product taxonomies, represent that structure explicitly as a graph rather than forcing it into a flat space. The geometry is no longer being fought, it is being avoided.

Domain-tuned embeddings. Fine-tune the encoder on hard negatives drawn from the organization's own corpus, meaning near-synonym pairs that are close in meaning but wrong as answers. Results show a measurable narrowing of the gap, though not its removal.

Each of these patches a specific symptom and none resolves the contradiction underneath, which is worth admitting plainly.

The larger research question

If different geometries represent different things well, the natural follow-up is whether the shape of the representation space could adapt to the query or the corpus and not be fixed beforehand.

The Symmetry review points that way and frames it as symmetry breaking: picking Euclidean, hyperbolic or a mixed metric means picking which semantic symmetry to give up. Its proposed direction is hybrid geometric space learning, where a system adjusts its metric structure according to the relationship being encoded at each point in the corpus and does not commit to one global geometry for the whole index. This is a direction proposed in a review paper, not a shipped technique with benchmarks behind it. It is a hypothesis to watch and not a foundation to build on today.

What it does usefully change is the question to ask. Instead of which embedding model is best, the question becomes what shape the semantic relationships in the data actually take, and whether the retrieval architecture has any mechanism to represent that shape.

What to take away

In production, high cosine similarity and correct retrieval usually line up closely enough that the distinction is invisible. Design effort belongs on the cases where the two drift apart unnoticed, with vectors near each other thanks to a shared topic and not a shared answer.

A better embedding model will not fix it, and no single architecture will either. A flat geometric space, whichever one is chosen, encodes an implicit bet about which relationships matter most in the corpus, and that is why hybrid retrieval, reranking and structure-aware indexing exist: nobody can place that bet without hedging. Embedding models are not bad. A geometric space was asked to do semantic reasoning it was never fully built for, and it does an impressive job right up to the moment it does not.