Home / Articles

Retrieval Over an Engineering Wiki: BM25, Token-Safe Dense Windows and RRF

Evaluation-first retrieval for a technical wiki: exact-evidence targets, a BM25 baseline, a MiniLM truncation bug, token windows and reciprocal rank fusion results.

Publisher: REACTAPP.TOP7 min read1701 words

This article is published in English.

Sources: Building RAG from the Corpus Up, Not the Model Down

One idea drives everything in the retrieval stage of this evaluation-first RAG project, which runs over a large engineering wiki:

What gets matched and what gets returned need not be the same thing.

Chunking had already produced small, bounded search records, with each original parent chunk kept intact. That raises practical questions. Should search be keyword-based or embedding-based? Should a matching child go straight to the language model? How many results should come back? And how can those choices be compared without leaning on a handful of queries that happened to look convincing?

The wiki is private, so the examples here are synthetic and corpus details are sparse. The implementation choices and measured results are real. The relevance labels are still exploratory and shouldn't be mistaken for production ground truth.

An evidence contract comes first

Not every search record counts as authoritative evidence. An ordinary heading chunk can serve as both the matching unit and the returned unit. A recursive child, a JSON fragment or a code window is different. It exists so search can find a larger structure, and it must not stand in for that structure.

Every search record therefore resolves to a retrieval target:

ordinary chunk  -> exact chunk
recursive child -> exact parent chunk
JSON locator    -> exact source document
code locator    -> exact source document

This mapping runs before final ranking. When two overlapping children of the same parent both match, they collapse into one target, so two result slots are never spent on fragments pointing at the same evidence.

Hydration runs after ranking. The retriever takes the target ID and loads the original parent chunk or source document, keeping its content, source path, line range and hash. The small child stays around as match lineage but isn't offered as the full answer context. The contract in a sentence: search over optimized locators, rank unique targets, return authoritative evidence.

Starting with a lexical baseline

Dense retrieval often gets treated as the default for RAG, yet a technical wiki is full of terms that lexical search handles without effort, such as API names, config keys, paths, error codes, shell commands, house acronyms and service names.

BM25 went first because it is fast, interpretable and hard to beat on exact identifiers. The tokenizer keeps terms like api/v2, config.yaml and error-code whole instead of stripping the very characters that make them distinctive. Each record is indexed together with some surrounding context:

document title
heading path
exact record content

Title and heading path matter because a short fragment can be ambiguous alone. A line such as "Restart the worker" says more once the index also sees "Operations > Troubleshooting > Recovery". That context only feeds matching and never gets spliced into the authoritative source content.

BM25 gave a strong, understandable baseline, and it exposed a property of the evaluation set. Questions written from source documents often reuse the source's own vocabulary, which favors lexical search. A BM25 win on this set therefore can't be assumed to carry over to future user queries.

Dense retrieval and a quiet truncation bug

The dense baseline used sentence-transformers/all-MiniLM-L6-v2, which yields normalized 384-dimensional embeddings, so cosine similarity reduces to a dot product. The first version embedded each search record as a single string. Chunking had already enforced a word budget, which seemed safe.

It wasn't. Chunk limits were counted in approximate words, while MiniLM accepts at most 256 model tokens. Paths, markup, JSON and identifiers have no stable ratio of words to tokens, so a record inside the chunking budget could still overflow the model. Sentence Transformers truncated such records without announcing it. Text near the end of a long record never reached the encoder. The source existed and the chunk was indexed, yet a query could miss because the relevant sentence had been cut before encoding.

The lesson: a chunking limit is not an embedding limit unless both are measured with the selected model's tokenizer.

Token-safe windows

The corrected dense index calls the model tokenizer directly. For each record, a capped slice of the token budget goes to the document title and heading path, and the remainder holds content. Long content turns into overlapping token windows with 32 tokens of overlap. After composing the final text, it is tokenized again to confirm it fits within 256 tokens.

search record
  -> contextual prefix
  -> tokenizer-aware content windows
  -> normalized embedding per window
  -> collapse matching windows to one retrieval target

The full experiment produced about twice as many dense windows as source search records. That is expected. Windows widen matching coverage but add no new returnable evidence, because several windows still resolve to the same parent chunk or source document.

Three units are easy to confuse here:

  1. Source unit: the original document or exact parent chunk.
  2. Search unit: a structure-aware child or an ordinary chunk.
  3. Embedding unit: a window of the search text sized to fit the tokenizer.

Only the first counts as evidence. The search and embedding units merely point to it.

Evaluating beyond "these look good"

Twenty source-derived questions were written to cover procedural, conceptual, identifier-heavy and cross-section patterns. Each lists one or more relevant retrieval target IDs. The labels are deliberately flagged as pending review. Source-derived questions exercise the pipeline well, but two biases come with them. Wording that echoes the source tilts toward BM25, and the first labels may overlook other passages that plausibly answer the question. So the scores serve as comparative evidence for this dataset and not as a production benchmark.

The operating point was five returned passages, with three metrics:

  • MRR@5 shows how early the first relevant target appears.
  • Recall@5 shows how much of the known relevant evidence lands in the top five.
  • nDCG@5 shows whether relevant targets cluster near the top.

The first comparison produced these numbers:

  • Contextual BM25: MRR@5 0.4875, Recall@5 0.5583, nDCG@5 0.4533.
  • Windowed MiniLM: MRR@5 0.1392, Recall@5 0.2500, nDCG@5 0.1593.
  • Hybrid with reciprocal rank fusion: MRR@5 0.5308, Recall@5 0.7583, nDCG@5 0.5480.

BM25 beat dense retrieval clearly on this question set. That doesn't make embeddings useless. The two retrievers failed in different ways, and that left room for fusion.

Hybrid retrieval without a hybrid index

The BM25 and dense indexes stay separate, and their ranked outputs are merged with reciprocal rank fusion. For a target d, the fused score is the sum, over retrievers, of 1 divided by (k plus that retriever's rank for d), with k set to 60 in this experiment.

Each retriever supplies its top 10 unique targets, fusion merges the ranks, and the five best hydrated passages are returned. Lexical and dense scores never need calibrating against each other, because fusion looks at rank positions and not raw score magnitudes.

Recall@5 rose from 0.5583 with BM25 alone to 0.7583, and first-hit ranking and nDCG improved too. On this exploratory set the complementary dense matches helped even though dense retrieval was weak by itself.

The implementation is a coordinating retriever and not a third physical index:

query
  -> BM25 top 10 targets
  -> dense top 10 targets
  -> target-level RRF
  -> top 5 targets
  -> exact parent/source hydration

It also records each retriever's rank and the matched locator. That lineage makes any result explainable, showing whether BM25, dense retrieval or both found it, and which child or embedding window led to the authoritative target.

Diversity that backfired

Hybrid output sometimes held several passages from one document, which looked redundant. A maximum-per-document selector was tried. The output looked more varied but retrieval quality dropped. In procedural or deeply structured documentation, several sections of one page may all be required. Document diversity was working as a visual preference and not as a relevance signal. The selector stays available for later experiments but is switched off by default.

The distinction to keep: repeated documents are not repeated evidence. Collapsing children that resolve to the same target is correct. Dropping distinct relevant sections merely because they share a document is a separate decision that needs task-specific evidence.

Guarantees of the final path

  • BM25 and dense indexes cover the same contextual search records.
  • Dense records are windowed using the model's real token limit.
  • Child and window matches collapse into unique retrieval targets.
  • Fusion combines ten candidates from each retriever with no score calibration.
  • Five exact parent chunks or source documents come back.
  • Original content and hashes stay authoritative.
  • Per-retriever ranks and matched-locator lineage remain inspectable.
  • Document diversity is applied only when evidence shows it helps.

The retriever is intentionally plain. It coordinates existing indexes, fusion and hydration, without hiding behavior behind a framework abstraction or adding another data store.

What is still missing

The system can now take a query and return five ranked passages of exact evidence, say where each came from and compare retrieval choices using repeatable metrics. That is still not a complete RAG system. The next stage has to decide how passages are assembled into a model context, how citations are required, how conflicting or insufficient evidence is handled and when the system should abstain. Retrieval metrics also need linking to answer-level faithfulness and citation correctness, so generated answers keep a clear line between what the wiki states and what the model merely knows.