Fifty Years of Retrieval Research Hiding Inside Your RAG Pipeline

A chunking spiral that fixed nothing

Imagine building a document pipeline for a finance client and losing several late nights to chunk sizes. The sweep starts at 256 tokens and climbs to 512 and then 1024, circles back to 512 with a 64-token overlap, and ends at 384 because a forum thread vouched for it. A notebook holds a small eval harness, and two chat assistants sit in other tabs. One is quizzed about chunking strategy as if it were a senior engineer, and its answers get checked against whatever the internet published that month. The work feels like exploring a frontier, and each round ends with a winner and a chart.

Then the corpus grows. The more documents go into the vector store, the worse retrieval gets. Something that looked sharp on a small sample turns mushy at scale and returns passages that sit close in vector space and are useless in fact. The natural reaction is to tune chunking harder, because chunking is the knob everyone knows how to turn. Days go by, with each run moving some number and each movement mistaken for progress. None of it touches the real problem.

Then a teammate glances at the config and asks, with no irony at all, "what's BM25?" There's no clean answer ready. BM25 is old, and it's vaguely "just keyword search", the thing everyone supposedly moved past. What the moment really shows is that the whole effort has been standing inside a fifty-year-old conversation without learning the names of anyone already in the room.

That's the confession behind this piece, and it applies broadly. Nearly every step of "building RAG" already has a name, and most of those names carry dates older than the people who sell the ideas back with a pitch deck. Those sellers mostly don't know the lineage either. The "we" and "the field" used below point at practitioners in general, and every one of these mistakes is a common one.

Others have noticed the pattern. A longtime search engineer from Vespa wrote in 2024 that hybrid retrieval and ranking pipelines let you combine signals from unsupervised methods such as BM25 with supervised neural rankers. Pulling documents to stuff into a prompt is information retrieval, and that work stretches far beyond a single cosine score into query rewriting, fusion, reranking and phased retrieval. A veteran relevance engineer has made a similar point repeatedly: retrieval didn't die, it went underground and kept quietly running the world.

Both observations hold up, though "RAG is just information retrieval" is a slogan, and slogans cost nothing. What changes working habits is reading the receipts. What follows is a map of them, one that does not begin in 2020 with the first fifty years ripped out.

Chunking, embeddings and meaning

Start with chunking, where the embarrassment began. Breaking documents into passages rather than treating them whole is a decades-old idea. Passage retrieval was already a research topic in 1993 (Salton, Allan and Buckley), and Callan examined passage-level evidence the year after. By 1997 Kaszkiel and Zobel were publishing "Passage Retrieval Revisited", so the subject was old enough to need revisiting. Their experiments on passage size and segmentation found that simple fixed-length passages performed surprisingly well, which is exactly what a night of chunk-size sweeps re-derives with a worse method.

TextTiling, from Marti Hearst in the same year, places boundaries where the topic really changes and ignores token counts. Today's "semantic chunking" vendors resell that notion, and the underlying paper predates most of their founders' first computers.

Next the embedding, which feels like the genuinely modern step that sets people apart from the keyword crowd. Representing documents and queries as vectors and scoring them by cosine goes back to Salton, Wong and Yang in 1975, the vector space model. Giving rare terms extra weight, a job an embedding model now absorbs into its parameters, was Karen Spärck Jones in 1972. And the step usually treated as the heart of it, matching on meaning instead of literal words so that vocabulary mismatch stops mattering, is Latent Semantic Analysis, published by Deerwester and colleagues in 1990 using an SVD of the term-document matrix to surface hidden structure.

Replacing the SVD with a transformer let plenty of people decide the idea belonged to them. The transformer is a real upgrade rather than a cosmetic one, because it reads each word in light of the words around it, and a single static LSA vector could never do that. The principle underneath stayed put: meaning outranks surface form.

Fusion, query rewriting and reranking

Hybrid search is the current favorite, where lexical and dense scores get blended and everyone feels smart. The blending method shipped in Elasticsearch, OpenSearch, Weaviate and Azure AI Search is reciprocal rank fusion, introduced by Cormack, Clarke and Büttcher at SIGIR 2009. One detail in that paper deserves more attention than it gets: a trivially simple fusion using a constant of 60, never tuned, outperformed the trained learning-to-rank systems it was compared with. An untuned default beat models that people had trained. Combining results from several engines was already standard practice in 1994, when Fox and Shaw used CombSUM and CombMNZ for it.

Query rewriting is where a coffee cup might get set down hard. In HyDE, a model invents a fake answer document and the system embeds that in place of the question, on the reasoning that a plausible wrong answer lands closer to the right documents than the question itself. It is a neat trick, and it is also Rocchio from 1971.

Rocchio's relevance feedback shifts the query vector toward documents that seem relevant and away from those that do not. Croft and Harper's 1979 pseudo-relevance feedback performs the same shift without any human input, simply presuming the top-ranked results are good enough to learn from. Seen this way, HyDE is neural pseudo-relevance feedback, the same maneuver pushed through a transformer many years on. Its paper does not cite Rocchio, which shows the forgetting is not limited to tutorials and forum threads and sometimes reaches the papers themselves.

The older work had also put a name on the failure that haunts HyDE-style systems in production: query drift. When the top results are poor, expanding the query from them makes matters worse, which is why veterans ran feedback a single time and then stopped. Agentic loops now repeat it over and over, and then everyone acts surprised when retrieval drifts off into unrelated territory.

The remaining items follow the same pattern. A cross-encoder reranker added after first-stage retrieval to tidy the top results is a two-stage cascade. Cascade ranking was formalized by Wang, Lin and Metzler in 2011, and RankNet gave learning-to-rank a neural ranker by 2005. A pipeline that runs BM25 and then a cross-encoder, shipped several times in one year, is a standard two-stage cascade under another label.

Evaluation is where it hurts most

Evaluation is the area where the lesson stings hardest, because it is handled worst. The practice of freezing a corpus, writing test queries, gathering human relevance judgments and scoring systems offline is the Cranfield paradigm, dating to Cleverdon and colleagues in 1966. NIST's TREC scaled it up starting in 1992, and Voorhees and Harman documented it in a 2005 book. Pooling, which allows judging results without reading the whole collection, came out of that work decades ago, as did meaningful metrics such as MAP and nDCG.

A typical RAG "eval" is a tiny, undocumented cousin of that. No relevance judgments exist, an unvalidated LLM acts as judge, and the test set was quietly put together out of queries that happened to work already.

Classic IR has an unflattering name for that last habit: sampling on the dependent variable. Building the eval from queries known to retrieve well guarantees a pass, and then the system ships and fails on everything nobody thought to try. The failures get labeled "hallucination" because no instrument points at the actual cause. The rigged set still has some value, since it flags real regressions on queries people care about, but it says nothing about the questions nobody asked, and those are the ones that fail in production. In the finance pipeline it happened on the first question from outside the set, such as a figure above a threshold or a date range, the everyday shape of finance queries nobody had tested.

The numbers are harder to argue with. Consider the assumption that sent people tuning chunk sizes when they should have been running a baseline: that dense retrieval beats lexical search by default. The BEIR study by Thakur and colleagues in 2021 pitted ten systems against eighteen datasets and produced a finding worth printing inside every RAG repo. Performance on the training domain says little about performance elsewhere. BM25 lagged neural models by seven to eighteen points on in-domain data, then outperformed most of them once the data came from a different distribution. The strongest dense bi-encoder won on only eight of the eighteen datasets.

A niche finance corpus is precisely the specialized, out-of-distribution material that BEIR cautions about. The assumption that dense would still win was so complete that nobody ran the lexical baseline, even though the paper recommending it had been public for four years.

A caveat is due, since the frontier keeps moving. Some commercial leaderboards now report dense retrieval beating BM25 by fifteen to twenty-five percent on BEIR. Possibly, but those figures come from vendors under vendor-chosen conditions and merit the doubt owed to any benchmark run by its seller. Peer-reviewed work still lands on the dull, correct conclusion: BM25 is a strong baseline and hybrid is the safe default. If that changes, it should be shown by somebody with nothing to sell.

A sadder, older habit is knocking down weak baselines and calling the result progress. Jimmy Lin's 2018 paper, with the delightfully cranky title "The Neural Hype and Comparisons Against Weak Baselines", argued that many neural retrieval victories were victories over a BM25 setup nobody had taken the trouble to tune. A follow-up in 2019 surveyed years of papers on one standard test collection and found no upward trend in effectiveness; the best published numbers were ten years old and nothing newer approached them. So when a RAG benchmark shows vectors flattening "keyword search", the first request should be for the BM25 configuration. The commonest outcome in this field is an effective old method losing to a careless copy of itself.

Position bias, rediscovered

Position bias is the finding that is funny in the wrong way. In "Lost in the Middle" (Liu and colleagues, 2023), models used information best when it came at the beginning or end of the context and worst in the middle, tracing a U-shaped curve. The figures are blunt. In multi-document question answering with twenty or thirty retrieved documents, GPT-3.5-Turbo did worse with the answer in the middle than its 56.1 percent closed-book score, so adding retrieved context left it worse off than answering from memory. In a synthetic test the weakest models dropped under forty percent with the target mid-input, and the U shape appeared across GPT-4, Claude and several open models. The field treated this as news about an exotic new machine.

Joachims and colleagues had run an eye-tracking study on how people read ranked search results in 2005 and followed up in 2007. Attention pooled at the top of the list, and users kept clicking the top link even when something lower was more relevant. The literature called it trust bias. The order in which evidence is presented changes how it gets used, whether or not the evidence is any good, and that was known about humans eighteen years before anyone "found" it in models.

The parallel is imperfect, and claiming it was exact would commit the same sin being criticized. Classical position bias is monotonic, strongest at the top and dwindling down the list, while the model effect is U-shaped with a middle dead zone and a late bump from recency. The shapes differ even though the problem family is shared. The model version is also unstable: later work, some from 2025, shows the middle penalty shifting as inputs approach the context limit instead of vanishing. That is another reason not to cite "Lost in the Middle" as scripture. What lasts is the point that ordering distorts use, and the precise curve is a moving target, which argues for measuring over memorizing.

Where the old maps run out

Giving ground makes the takedown less satisfying, but stopping at the receipts would produce a clever takedown, which is a way of being wrong at length. Old maps do run out, and the places where they fail matter most.

The deepest break concerns who or what does the reading. For half a century a person scanned a ranked list, so relevance meant whatever a human could judge at a glance. A generator does the reading now, and recent research states the change plainly: this is search for an AI, not for a person. Relevance therefore depends on its environment and is hard to specify beforehand.

The change overturns old assumptions completely. A 2024 SIGIR paper by Cuconasu and colleagues, "The Power of Noise", ran the experiment. One result is expected: the retriever's top-scored documents degraded output when they were close in topic yet lacked the answer. The other is startling: random, unrelated documents lifted accuracy by up to thirty-five percent. Cranfield treated a document's relevance as its own independent, additive property. In RAG, a passage's usefulness hinges on everything else in the context and on how the generator combines it. The basic unit of relevance shifted, and most metrics still count the previous unit.

Another failure mode has no classical ancestor, because old systems couldn't commit it. Classical IR could rank badly, but it couldn't lie. It returned documents that existed, and it never wrote a fluent, confident paragraph that those documents flatly contradict. Faithfulness, meaning whether the generated text is actually supported by what was retrieved, is a new axis. Tools like RAGAS, which appeared in 2023, measure something the whole earlier field never had to name.

Agentic retrieval is only partly new. One search call is ancient. What differs is a model that breaks a question apart, issues its own sub-queries, reads the results and revises its plan, which amounts to an autonomous control loop around ordinary retrieval. That loop is the innovation. Khattab's DSP and DSPy projects are the serious treatment, and it fits that they come from someone fluent in the older literature.

The scoring side needs rethinking as well. nDCG, MAP and MRR rate a ranked list for a person reading down it, and they are silent on whether the final generated answer is correct, complete and grounded. Importing the old metrics unchanged does not finish the job.

The researchers are still around

The people who built all this didn't retire to a beach. Several of them are quietly folding RAG into the evaluation tradition that RAG spent two years ignoring.

Jimmy Lin maintains Anserini and Pyserini, tooling that reduces a well-tuned BM25 to one command, which removes the weak-baseline excuse. After the cranky "neural hype" papers he published "The Neural Hype, Justified! A Recantation" in 2019, conceding that against strong baselines the BERT-era neural methods did deliver gains in low-resource settings. He had said the field was overhyped, examined stronger evidence and reversed himself in print, and was still partly right about the hype. That attitude is worth adopting from both directions, with veterans moving toward the new work and newcomers reading back far enough to see what they are updating from.

Bhaskar Mitra and Nick Craswell wrote a neural IR primer in 2018 that set neural models firmly within the classical tradition and treated BM25 as a baseline owed respect. From 2019 they led the TREC Deep Learning track, evaluating the entire neural era on the pooled, reusable collections that Cranfield and TREC had refined for decades. The revolution was graded using the old field's instruments, by the people who made them, and it passed. The rigor had been portable from the start, if anyone wanted it.

Ellen Voorhees and Donna Harman edited the standard work on TREC evaluation. Voorhees spent years on an awkward fact: human relevance judgments conflict with each other, which is the very difficulty facing anyone replacing human assessors with an LLM. When the IR community considered that swap, it argued the case in print, and named researchers warned that it might silently corrupt the science. Pilot data showed model judgments tracking human ones with a Kendall's tau near 0.86. That looks comforting, but one correlation reveals nothing about where the two disagree, and the disagreements that count are borderline documents that decide the winning system.

A track now exists for this. In 2024 NIST started a TREC RAG track with 301 topics over a deduplicated MS MARCO collection of 113 million passages. Those who see evaluation as running pytest should examine the dull labor that goes into a test set worth trusting. The track grades nugget recall, which brings Voorhees's old nugget method into the LLM era, and citation support, which checks that generated claims are backed by retrieved passages. In 2025 the queries moved from short keyword strings to long, messy, multi-sentence narratives, closer to how real questions arrive. The establishment was not disrupted out of existence. It set up a track and invited RAG in, which is a very TREC move.

What changes when the history gets read

Retrieval is not dead, and RAG did not change everything. Anyone selling either story deserves suspicion, since both let a person skip the work. The truth is smaller and more useful. The field gained a remarkable way to consume retrieved text and, in the rush, discarded fifty years of earned knowledge about how to retrieve it well.

That knowledge can be picked up again at any time. The citation graph is public, the benchmarks are public, and the researchers are alive and, by reputation, pleasantly surprised when someone younger turns up with an informed question.

The practical changes are boring on purpose:

• Tune the BM25 baseline before touching the embeddings.
• Build eval sets from the queries that cause worry, and skip the ones that already work.
• Treat the related-work section as the place where a field keeps its memory, and follow citations backward until they stop.

None of this delivers the buzz of tuning a chunk size. A baseline that is expected to win lacks the excitement of re-deriving a frontier, and that missing excitement has a real cost. It also separates a pipeline that demos nicely from one that holds up on a corpus nobody scraped.

The teammate who asked about BM25 was not behind the times. He had received the same map as everyone else, a version that begins in 2020 with the first fifty years ripped away. The cure is to locate the missing pages, and the fortunate part is that they were never lost. People simply stopped reading them.

The original papers, grouped

Reading at least one of these beats taking anyone's word for it.

Passages, chunking and embeddings:

• Salton, Allan and Buckley (1993), on passage retrieval in full-text systems, years before the web had a search engine.
• Callan (1994), "Passage-Level Evidence in Document Retrieval", which compared paragraph and fixed-window passages inside a real retrieval system. The "chunking strategy" has a parent.
• Kaszkiel and Zobel (1997), "Passage Retrieval Revisited", whose passage-size experiments found fixed-length passages effective, a result any eval harness will rediscover if allowed to.
• Hearst (1997), "TextTiling", on splitting text where the subtopic shifts. Every semantic chunking pitch has an older citation.
• Salton, Wong and Yang (1975), the vector space model, which wrote down in 1975 the operation a vector database performs at query time.
• Spärck Jones (1972), on term specificity, the origin of idf, which embedding models now learn for free.
• Deerwester and colleagues (1990), "Indexing by Latent Semantic Analysis", whose SVD over a term-document matrix is the idea many people think they invented in 2020.

Fusion, feedback and reranking:

• Cormack, Clarke and Büttcher (2009), reciprocal rank fusion, the hybrid-search method inside several major engines, with an untuned constant of 60 that beat trained methods.
• Fox and Shaw (1994), combining multiple searches, with CombSUM and CombMNZ already routine.
• Gao, Ma, Lin and Callan (2022), the HyDE paper, an elegant fake-answer trick that is in spirit neural pseudo-relevance feedback.
• Rocchio (1971), the original relevance feedback move, nudging the query toward what looks relevant.
• Croft and Harper (1979), pseudo-relevance feedback, which also named query drift.
• Wang, Lin and Metzler (2011), the formal cascade ranking model behind BM25-then-cross-encoder pipelines.
• Burges and colleagues (2005), RankNet, proof that learning-to-rank had a working neural ranker in 2005.

Evaluation:

• Cleverdon, Mills and Keen (1966), Cranfield II, the sixty-year-old template of corpus, queries, judgments and offline scoring.
• Voorhees and Harman (2005), the TREC evaluation book, about testing retrieval without fooling yourself.
• Järvelin and Kekäläinen (2002), the origin of nDCG and its justification.
• Voorhees (2000), on how much human judges disagree, the exact problem behind swapping in an LLM judge.
• Faggioli, Dietz, Clarke and colleagues (2023), perspectives on LLMs for relevance judgment, where the warning about corrupting the science comes from.
• Mitra and Craswell (2018), the neural IR introduction that sets neural models inside the classical tradition.
• Craswell and colleagues (2019), the TREC 2019 Deep Learning track overview.

Empirical results:

• Thakur and colleagues (2021), BEIR, the receipt showing BM25 beating most dense models off-distribution.
• Lin (2018) and Yang, Lu, Yang and Lin (2019), on weak baselines and the missing upward trend in effectiveness.
• Lin (2019), the recantation, a critic changing his mind in print.
• Liu and colleagues (2023, published 2024), "Lost in the Middle".
• Joachims and colleagues (2005, with a 2007 follow-up), on clickthrough as implicit feedback and the trust bias it exposes.
• Veseli, Chibane, Toneva and Koller (2025), on positional biases shifting as inputs approach the context window limit.

What's genuinely new:

• Cuconasu and colleagues (2024), "The Power of Noise", where random unrelated documents raised accuracy.
• Es, James, Espinosa-Anke and Schockaert (2023), RAGAS, tooling for measuring faithfulness.
• Khattab and colleagues (2022 and 2023), Demonstrate-Search-Predict and DSPy, the serious version of agentic retrieval.
• The TREC 2024 RAG track at NIST, with 301 topics, 113 million passages, nugget recall and citation support.