Why RAG Misses the Page That Holds the Answer: Bread, Yeast and Vectors
Similarity search in RAG can return pages on the right topic and still miss the one with the answer. Why tables and top-k make it worse, and what to test first.
This article is published in English.
Picture a loaf of bread that comes out of the oven flat and dense. You ask an AI assistant that has ingested your whole cookbook why it happened. The reply arrives instantly, with total confidence: check your kneading technique, check the oven temperature. Everything it says concerns bread, and none of it is correct.
The cookbook did contain the answer. A page headed "Activating yeast" states that water hotter than about 50°C kills yeast. The water was too hot, the yeast died, nothing rose, and the loaf stayed flat. That page was never opened. The reason is worth understanding, because the same thing is quietly happening inside a great many company AI systems.
How these systems work
The technique is retrieval-augmented generation, or RAG. A model can't memorize all your documents, so the system fetches the relevant passages and lets the model read them before it answers.
One detail matters later. The system doesn't keep whole documents. It slices them into fragments of a few hundred words, called chunks, and searches those. In a cookbook, a chunk is roughly a page.
A useful picture is a librarian working with an expert. Your question goes to the librarian, who walks the shelves, collects several pages that seem on topic and passes them through a slot to an expert shut in a room. That expert may use only those pages and has no way to request more.
How many pages get passed along? Often five by default, as in Amazon Bedrock Knowledge Bases and Azure OpenAI On Your Data, while LangChain starts at four. Raising the count is the first fix most people reach for. Each platform labels it differently: numberOfResults in Bedrock, topNDocuments in Azure OpenAI On Your Data, and k in LangChain and in Azure AI Search itself. They all mean how many fragments the model gets to read.
When the librarian chooses well, the expert looks brilliant. When the choice is poor, the expert doesn't announce that the pages are wrong. It answers anyway, fluently, confidently and incorrectly.
How the librarian picks
Everything else follows from this step. The system turns each chunk into a list of numbers describing what the chunk is about, and does the same for your question. It then returns the chunks whose numbers lie nearest to the question's.
Two names come up constantly. A chunk's list of numbers is called an embedding, and the yardstick for nearness is cosine similarity. You can skip the math, since the idea is simply how close in meaning two things are.
The flaw is visible in the mechanism
Think of each chunk as an arrow aimed in some direction. The similarity score reflects the angle between two arrows: a small angle suggests alike, a large one suggests unrelated. Because the formula divides out the length of each arrow, only the heading of a text survives. Strength, precision and emphasis are all thrown away.
You can demonstrate the consequence in about a minute. Take pairs of sentences that differ by a single word, where that word carries the whole instruction, such as "do" versus "do not". To the search, each pair looks almost the same. If it retrieves the wrong member of a pair, the final answer isn't vague. It's confidently backwards.
Run a quick check on your own material before relying on a system. Pick two sentences that differ only in a negation, a version number or a threshold, embed them and compare the scores. This convinces more than any published benchmark because it uses your corpus.
To be fair, this approach isn't keyword matching. It compares meaning, so synonyms are no trouble: ask about a car and it finds text on automobiles. People often assume better wording is the cure, but wording isn't the problem.
What the score measures is whether a chunk is about the same thing as the question. What it doesn't measure is whether the chunk contains the answer. Most of the time the two coincide and nobody notices. A question about when the Eiffel Tower was built is answered by a chunk about the Eiffel Tower, which is easy.
Now return to the bread.
When the answer lives in a different subject
The question is about bread. The page holding the answer is about yeast. Those are different subjects, not different words for one subject, so clever phrasing can't bridge them. What connects them is a causal sequence:
water too hot, yeast dies, no gas, no rise, flat loaf
An experienced baker walks that sequence without a second thought. A similarity search can't cross it at all, because crossing requires reasoning, and the search only measures closeness.
It gets worse. You asked precisely because you didn't know the cause. Had you known to look up yeast temperatures, there would have been no need to ask. These systems shine on questions you could have answered yourself.
Why fetching more pages helps only a little
Raising the setting from five to fifty looks like the obvious repair, and the yeast page might now scrape in. But the ranking hasn't changed. The page remains buried because it's still about another subject. The net is wider and you're hoping.
There's a bill, too: more text means higher cost, slower replies and extra near-relevant material cluttering what the model reads, which, as shown below, actively worsens results.
Tables are lost before search even starts
A second loss occurs earlier, when documents are ingested. Before anything can be searched, a document is converted to plain text. Prose survives the conversion. Structure often doesn't.
Imagine a table listing proofing times, with one column for each bread type. Flattened to text, the numbers survive but the columns vanish. Ask how long rye needs, and the stored text offers no link between the figure of 45 minutes and rye as opposed to sourdough. Only the table's columns expressed that pairing, and they didn't survive the conversion. A flattened table can therefore hand the model a list of loose numbers and names.
A cookbook shrugs this off. A specification sheet, pricing table, compatibility matrix or configuration reference doesn't, since in those documents the meaning sits in the layout and not in the sentences.
Both big clouds now offer better parsing: Bedrock has a foundation-model parser and Azure has Document Intelligence layout extraction. They help a lot, but they don't eliminate the problem, and they aren't what you get by default.
Measured evidence
A benchmark named BRIGHT targets questions where reasoning, not topic matching, leads to the right document. The retrieval models that reach about 59 out of 100 on ordinary lookups fall to about 18 on those questions. Nothing differed except the kind of question. More compute doesn't rescue them either: among the thirteen models tested, the best result on the hard set was 24.
The dangerous part
Had the system answered "nothing found", this would be a small annoyance. It never does. A search engine orders results by closeness and doesn't turn any away. When your documents lack the answer, you still receive results that look exactly like a success, and no cutoff score exists where the system admits it found nothing useful.
Work presented at ICLR 2025 examined what follows. When a model received related material that didn't actually contain the answer, its wrong answers climbed from roughly 10% to 66%, worse than providing nothing. The models also grew noticeably less willing to say they didn't know.
Material that is partly relevant does more harm than none, since it creates false confidence. A flat loaf is cheap to bear. The same mistake inside a compliance review, a medical question or an architecture choice is not, and that is why this failure deserves attention.
What to do
Three steps help, roughly in order of payoff:
- Use realistic questions. Teams usually test with queries written by the authors of the documents, using the documents' own words. Have colleagues who haven't read the material write the questions instead, and the weak spots surface quickly.
- Measure retrieval on its own. Before judging whether an answer was good, check whether the right chunk even came back. If it didn't, adjusting the model won't help, because the wrong end of the system is being fixed.
- Let the system search more than once. The lasting remedy is an AI that searches, reads the results, recognizes they fall short and searches again with better terms, as a person would. That's the line between a fixed pipeline and an agent, and the direction the field is moving. In the bread example, a first search for flat loaves could surface pages about kneading, notice that none mention rising failures, and try again with terms like proofing, fermentation and yeast temperature, which is how the right page would finally be found.
Conventional RAG struggles with deep technical questions for several reasons, and this gap between retrieval and reasoning is only one of them. Chunking that destroys meaning is another, and so is a system that passes testing yet fails in production. Fixing any of these starts with measuring where the right passage got lost, rather than guessing, and with assuming that the first retrieval attempt will sometimes miss.