RouteRAG: Teaching a Small Model When and How to Retrieve with RL
RouteRAG trains a small model with two-stage GRPO to pick passage, graph or hybrid retrieval, cutting retrieval turns and lifting multi-hop QA results.
This article is published in English.
Sources: RouteRAG: An RL Router That Teaches RAG When to Search
Pairing RAG with reinforcement learning has become a more common research direction, and RouteRAG is a clear example. The paper trains a language model to decide for itself how to retrieve, from text, from a graph or from both, and when to stop.
The gap it targets
A language model's knowledge is frozen at pre-training time. For questions that need external or fresh information, it often returns stale or wrong answers. Newer methods use multi-turn retrieval, where the model reasons, fetches new information, reasons again and repeats. Smaller open-source models struggle with that loop. They have trouble deciding when to search, how to phrase a query and how to analyze what came back. Many prompt-based pipelines quietly depend on large closed-source models with strong built-in reasoning and planning.
Graphs add a second layer of difficulty. They make relationships between entities explicit, which suits multi-hop reasoning, yet pulling and processing graph evidence is computationally costly. Most systems also treat graph retrieval as a single, fixed step in the pipeline and rely on heuristics, with no way to bring in more graph evidence as reasoning develops.
Retrieving more documents, or flipping between graph and text, isn't the cure. The model needs a learnable policy trained with reinforcement learning, one that actively chooses what to do next: retrieve from text, from a graph or from a mix, when to stop and how to trade accuracy against efficiency. Smarter retrieval beats more retrieval.
Retrieval as sequential decisions
RouteRAG frames multi-turn RAG as a sequential decision process. Instead of fixed, handcrafted pipelines or static instructions, it learns a policy in which special tokens inside the reasoning stream trigger retrieval actions.
At each step the model emits an action token. It can keep reasoning internally, start a search with <search>...</search>, or finish with <answer>...</answer>. Inside <search>, a tag picks the retrieval mode: [passage], [graph] or both, as in [graph][passage], which means hybrid retrieval.
The contrast with earlier work is the main thing to remember. Previous RL-based multi-turn RAG methods mostly interleave reasoning with passage retrieval and reward answer correctness alone, an outcome reward, with no explicit pressure on cost or efficiency. RouteRAG widens retrieval to passage, graph and hybrid modes. It trains in two RL stages built on GRPO. Stage 1 rewards correctness only. Stage 2 adds an efficiency reward derived from total retrieval time, which pushes the model toward selective retrieval while protecting answer quality.
The loop in detail
Step by step, the policy model πθ generates tokens, and it has at most B steps to work with. A <search>...</search> emission is parsed into a sub-query q′ plus a retrieval mode m ∈ {Passage, Graph, Hybrid}, and both go to the retriever R. Whatever evidence d comes back is enclosed in <information>...</information> and appended to the context for the following round. The loop stops as soon as <answer>...</answer> appears.
During training the budget is set to B=4 steps, and every retrieval yields 3 passages. That count is unrelated to the smoothing constant in the RRF formula below, which happens to share the same letter.
So the model isn't guessing whether to search or what to search for. It learns when to retrieve, how to word sub-queries and which mode to use.
Three retrieval modes
- Passage retrieval uses DPR-style dense search. The sub-query and corpus passages are encoded into a shared embedding space, and the top-k by similarity are selected.
- Graph retrieval builds a knowledge graph from the corpus with HippoRAG 2. For a query, personalized PageRank starts from the nodes related to it and surfaces evidence along multi-hop connections.
- Hybrid retrieval merges the two result lists with Reciprocal Rank Fusion (RRF). A document ranked high in either list gets boosted in the merged ranking, as the formula shows:
RRF(d) = ∑ over m ∈ {Passage, Graph} of [1 / (k + rank_m(d))]
Complementary evidence is therefore more likely to survive, since a document rated highly by either mode gets promoted and is less likely to be missed.
Training with GRPO
RouteRAG learns through Group Relative Policy Optimization (GRPO), which stabilizes training by comparing trajectories within a group and so lowers variance when rewards are sparse. The reward design is where the method gets its edge.
Stage 1, accuracy only. The model gets 1 if its answer exactly matches the ground truth and 0 otherwise. The goal is a solid base: learn to answer correctly and ignore cost for now.
Stage 2, accuracy plus efficiency. Only correct answers qualify for the efficiency reward. Total retrieval time t is centered on the batch average t_avg and normalized by T, a constant that keeps t and t_avg scaled within [0, 0.5], according to the paper.
R_efficiency = (t_avg - t) / T
So a right answer still loses points if it took longer than the batch norm. For stability, the training objective mixes clipped policy ratios with a KL penalty and a group-relative advantage Aᵢ. With the two-stage rewards on top, the policy tilts toward trajectories that are correct and also cheaper to retrieve, meaning faster and more selective, and doesn't depend on stumbling onto such habits by chance.
Evaluation
Testing covered five common QA datasets: PopQA, Natural Questions (NQ), HotpotQA, 2WikiMultihopQA (2Wiki) and MuSiQue. Two metrics were used, Exact Match (EM) and F1, with results split into simple QA (PopQA, NQ), multi-hop QA (HotpotQA, 2Wiki, MuSiQue) and the overall average of all five.
Multi-hop QA gets the clearest boost, and the overall average rises too, even though the backbones are fairly small: Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct. On multi-hop tasks, results approach those of graph RAG systems running on GPT-4o-mini and exceed a few. Note that the graph baselines use GPT-4o-mini, whereas text-based baselines share the Qwen2.5 backbone. The cost is some lost accuracy on simpler QA.
A case study shows the effect.
Prior to training, the model invents facts from memory: it claims Johnny Pemberton played Bo Thompson in That '70s Show and names Steven Molaro as creator. Afterwards it avoids these slips, splits the question into steps, fetches the correct evidence and arrives at Justin Spitzer as the real creator.
What stands out
RouteRAG folds the whole reason, retrieve and generate loop into one learnable policy. It pairs reasoning with decisions on what to retrieve (passage, graph or hybrid) and when to give a final answer within a step budget B. Optimized with the two-stage GRPO-based RL setup, it delivers sizable gains even with small backbones.
Its most valuable property isn't being one more RAG variant. The model learns, inside its policy and not through hardcoded rules, when to retrieve and when to answer under a step budget. As a result it trims retrieval rounds from 2.70 to 2.25 against the variant without the efficiency reward, with F1 unharmed.
Two caveats for real deployments
- Generalization. Results cover only 3B and 7B models, and graph retrieval always uses HippoRAG 2. How the routing holds up with bigger models or different graph pipelines is unknown. In production, shifts in graph quality or coverage could change the planner's behavior directly, and that hasn't been tested.
- Reward design. The efficiency term relies on batch-level average retrieval time. The authors say this lowers noise in the timing signal and steadies training. Whether it also nudges the policy away from slower graph retrieval wasn't tested directly, so it stays a plausible but unconfirmed risk, one that could hurt long-tail multi-hop questions where graph reasoning is essential. Rewarding efficiency only on correct trajectories, combined with GRPO's group-relative advantage, encourages selective retrieval without sacrificing answer quality, but nothing guarantees that a slow yet necessary graph lookup always gets chosen.
For reference, the paper is titled RouteRAG, with the subtitle about efficient retrieval-augmented generation from text and graph via reinforcement learning.