Eight RAG Best Practices Tested on 746 Pages of Manuals

A support bot built over three real PDFs, a 408-page EV owner's manual, a 329-page TV guide and the 9-page leaflet packed with a wireless mouse, became a testbed for eight popular RAG optimizations. The rule was strict: nothing gets adopted unless a measurement on that corpus, scored against a labelled answer key, says it worked. Literature and habit do not count.

The outcome was accuracy rising from 72% to 89% while prompt tokens per query fell by 14 to 57%. That came from four changes, not eight. Two of the dropped ideas actively hurt, and one shipped optimization turned out never to execute. The whole investigation cost about $0.015 in API calls.

Here is how each idea fared. The most valuable item on the list is not a RAG technique at all.

A bug bigger than any hyperparameter

Before tuning retrieval, check what really reached the index. Everyone confirms that the PDF parsed. The better question is what condition the text is in.

A free loop answers it:

SOFT = "­"    # U+00AD, the soft hyphen
for manual in manuals:
    print(manual.name, extract(manual).count(SOFT))

The car manual returned 2,272.

A soft hyphen is a typesetter's instruction to the renderer: break the word here if the line runs out. A PDF reader obeys it and hides it, so eyes never see it, but a text extractor passes it along raw. The page shows a clean word, while the index receives strings like ve, soft hyphen, newline, hicle.

In that manual, 999 distinct words were split in half, including, repeatedly, the most important noun in any car manual. Vehicle appeared 1,486 times as a whole word and should have appeared 1,580 times. Ninety-four occurrences were unreachable by search.

This damages every retrieval path simultaneously. Keyword search matches literal strings, and hicle is not vehicle, with no partial credit. Vector search fares worse than hoped, since embedding models break text into subword pieces and a fragment like hicle shatters into pieces with no meaning. Even a lucky hit pastes garbled context into the prompt.

The repair is one line, with a trap beside it. Justified text also breaks on real hyphens. A broken cus-tomer should rejoin as customer, but toll-free must stay hyphenated, and telling them apart needs a dictionary. There were 95 of those, and all were left broken, with the number written down. A known gap of 95 words is a very different artifact from a silent guess that quietly corrupts a corpus. No chunk size recovers a word the index never held as a word.

The best chunk size is a trap

Bigger chunks hold more context and so match more questions, but they also drag extra irrelevant text into every query, permanently. Overlap exists so that an answer sitting on a boundary is not cut in two.

Nine configurations were run against the answer key. Accuracy costs nothing to measure, while cost is what is paid for it.

Overlap never earned its place, though not in the way a first reading suggests. At 700 characters, dropping overlap gained 11 points, which looks damning until the other pair is checked: at 1800, overlap of 0, 100 and 200 all scored exactly 89%. One comparison says overlap is expensive and the other says it does nothing.

What accounts for both results is a statistic that almost stayed off the page. Across every configuration, even the smallest, each answer fit wholly inside one chunk, and none was ever cut at a boundary. Overlap was insurance against a problem this corpus lacks, so its measured value swung between zero and negative. Checking how often answers really straddle a boundary should precede paying that premium.

Then comes the trap. The best score, 89%, came at 1400 characters with 200 overlap. Taking it means sending 1,283 tokens per query instead of 750, which is 71% more on every query for the lifetime of the product. With 18 questions in the key, 89% means 16 correct and 83% means 15. The top configuration beats the cheapest by exactly one question.

So the winner should not be taken by default. Choose the cheapest configuration whose score cannot be told apart from the winner's, which here meant anything within 8 points. Six of nine configurations tied under that rule, and the cheapest of them won. Accuracy moved from 72% to 83% with a smaller token bill besides, the project's first free lunch, earned by declining to optimize the metric being measured.

A filter that cost 22 points

The corpus covers three products. A question about the mouse has no reason to search a 408-page car manual, since the other two documents only offer chances to be wrong. The plan was to detect the product and restrict search to that manual.

A classifier did it. It chose correctly 78% of the time in under 4 ms at no cost, a solid majority compared with searching three times as much text. Yet accuracy fell by 22 points.

The mechanism took embarrassingly long to see. A wrong filter does not demote the right answer. It deletes it. Without filtering, a confused search across all three manuals still surfaces the correct passage, perhaps at position 4 instead of 1, and a reranker can rescue it. With filtering, a misrouted question searches the wrong document, and the correct chunk is not ranked low because it does not exist in the candidate set.

A classifier that is right 78% of the time therefore does not cost a little quality at the margins. It installs a hard 78% ceiling over everything downstream, however good the rest of the pipeline is.

The remedy is not a more accurate classifier but one allowed to abstain: apply the filter only when confident, and search everything when the top two products score close. In a question like how do I pair it over Bluetooth, the word it carries a lot of weight, and the classifier is as unsure as anyone.

Or skip inference entirely

The cheapest router is a dropdown. For a tool of this kind, asking which product a question is about costs a single tap, frequently one the user is making anyway by opening chat from a product help page. That resolves instantly an ambiguity the classifier can merely guess at, and the filter is correct every single time.

The numbers are exact. The classifier misrouted 4 of 18 questions. Search without a filter found 15 answers, search with it found 11, and each misroute turned a hit into a miss. Take the misroutes away and the accuracy remains while the precision the filter promised arrives as well.

The optimization that never ran

This part would be tempting to delete for the sake of appearances.

The plan was to answer with a cheap 20B model when retrieval looked confident and to escalate to a 120B model when it did not. Wrong answers tend to cluster where retrieval was shaky, and the reranker already produces a confidence score, so the signal costs nothing.

The threshold was set to 0.0, which felt principled: a cross-encoder emits a logit, sigmoid(0) equals 0.5, and escalating below zero reads as escalating when the top chunk is more likely irrelevant than relevant. Printing the real distribution told another story. Observed top scores had a minimum of 3.24, a median of 8.29 and a maximum of 10.00.

The threshold sat below the lowest score ever seen, so it never fired, not once. The escalation policy was byte-identical to always using the cheap model, plus overhead, a row in the results table that measured nothing while looking like a finding. A zero threshold is not a neutral default. It is a claim about a distribution nobody examined, and every absolute logit threshold has that flaw, because the distribution shifts when the reranker or the corpus changes.

The fix was to derive the number. Run only the two endpoints for real, always cheap and always expensive, recording each question's score, correctness and cost. Every threshold in between is then a replay over rows already in memory, with zero extra API calls and an exhaustive sweep instead of a sampled one.

The sweep produced a surprise: every threshold only changes cost, and accuracy never budges. The diagnostic shows why. Among twelve questions, just one could be recovered, meaning the cheap model failed and the expensive one succeeded. That question scored 10.00, the top of the range, whereas questions the cheap model already got right had a median of 8.29. The only failure worth escalating was the one retrieval felt surest about, so no rule built on escalating when unsure can catch it at any threshold. The routing signal fails to predict the thing it was meant to predict.

A side effect surfaced too. The live run reported 92%, yet the replay over already-paid-for rows predicted 83%. The difference is not the threshold doing anything. It comes from the LLM judge scoring identical answers differently on a second pass, and spotting that avoids announcing a 9-point gain that is only noise.

The change that paid twice

Reranking was the cleanest win, and the reason is worth understanding as well as adopting.

A vector search embeds the question and every chunk on their own, through a model that never views them as a pair. A cross-encoder takes the pair as a single input and scores it directly, which is much more precise and much too slow to run over a whole corpus. The pipeline therefore fetches 25 candidates cheaply, re-sorts them and keeps the top 4.

Recall rose from 83% to 89%. The more telling number is where the answer landed. Before reranking, the right chunk was typically second or third, so five were sent to be safe. After, it was usually first, and four were enough. Settling for baseline accuracy allows going down to two chunks, for 57% fewer tokens.

Because ordering improves, fewer chunks reach the same accuracy, and prompt tokens are what gets billed. The change purchases accuracy and returns a discount alongside it. Its price is 73 ms of local CPU, compared with an LLM call of 1,252 ms, a gap nobody notices.

What carries over

• Inspect the input before touching the retriever. One cheap loop either uncovers something like the hyphen problem or confirms the ceiling is clear, which makes later spending safer.
• A config default is another person's measurement, or no one's. An overlap of 100 never helped and a 0.0 threshold never triggered, and in production neither would announce itself. The only symptom would be a bigger bill with no explanation.
• Consider how an optimization behaves in the cases where it errs, and not only how often it errs. A filter that misses 22% of the time but merely reorders results can be lived with. One that misses 22% of the time and makes the answer unreachable cannot.
• A rejected idea is a result. Four ideas left undeployed are four more things with no maintenance, monitoring, versioning or middle-of-the-night alerts.

One caveat applies. The answer key holds 18 verified question-and-quote pairs, so a single question is worth 5.6 points. The 22-point routing collapse involves four questions, which is enough to act on. The 11-point overlap result involves two, so it is directional at best. An answer key built on a team's own data should come before trusting anyone's numbers. The full run takes about five cents on a laptop CPU, with a hard budget ceiling that raises an exception instead of surfacing the figure on an invoice.