A 270M Model for Multi-Hop Questions: Accuracy Losses and Local Wins

A 270-million-parameter model was fine-tuned for one narrow piece of multi-hop reasoning and then compared with far larger models. It lost on every accuracy number. It's still useful, and the reasons are worth spelling out.

The job sounds modest. A multi-hop question comes in, and the model has to name the opening lookup for a retrieval system. Out of 95 questions kept aside for testing, the tiny Gemma-3-270M got that right 81 times, which is 85.3%. Both larger rivals scored higher. Even so, it fits on a laptop, costs nothing per call and works without a connection. One extra finding came out of the work: a fixed system prompt helps a model of this size far more than expected, and the gain can be put in numbers.

The setup

Multi-hop questions need more than one fact. Take "Were Scott Derrickson and Ed Wood of the same nationality?" Answering requires finding one person's nationality, then the other's, then comparing. HotpotQA, the dataset used here, consists of questions like this, each needing at least two facts that usually come from two different Wikipedia articles.

The target was the first step only, which is called Hop-1: the first lookup a system must perform before it can go find anything else. For a given question, the model writes a short YAML object with three parts. thought explains what to look up first, action is always the fixed value Lookup, and target_entity names the exact thing to search for. A correct output for the Derrickson question looks like this:

thought: "I need to find the nationality of Scott Derrickson first to eventually compare it to Ed Wood."
action: "Lookup"
target_entity: "Scott Derrickson"

The model retrieves no documents, runs no second hop and gives no final answer. It only decides what to look up first.

For comparison questions, the model could have emitted both entities and the comparison in a single output. The design deliberately avoids that. Each hop gets one atomic action, so a comparison question turns into three separate steps. Find entity one, find entity two, and only then compare the results. When something goes wrong, the failing step is easy to pinpoint. It also costs accuracy, because errors compound along a longer chain. At 90% per step, a three-step chain lands around 73% end to end. The trade is acceptable because a system that fails and says where is worth more than one that fails silently.

Training needed labeled examples of correct Hop-1 decompositions, and GPT-4o served as the teacher. Its prompt contained five worked examples and strict rules: the target must come straight from the question's own wording and never be a name the model works out by itself, and anything unknown goes into thought and not into target_entity. That prompt ran over HotpotQA questions, and a second GPT-4o call graded each output against HotpotQA's own gold article titles. 474 labels survived the check.

The teacher's rules matter more than they look. Forcing the target to reuse the question's own words keeps the small model from having to resolve names from memory, which is exactly where a 270M model would hallucinate. Putting uncertainty in thought keeps target_entity clean enough to feed straight into a search call.

The labels were split 80/20, giving 379 for training and 95 held out for every evaluation below. Gemma-3-270M was then fine-tuned on the 379 with supervised fine-tuning, meaning direct training on question and answer pairs and not on any reward signal.

How scoring works

Every figure below uses the same 95 held-out questions and two separate measures.

• Target correctness. Did the model pick the right thing to search for? A hit means target_entity equals one of HotpotQA's gold titles for that question (case ignored), or else a GPT-4o judge rules that both strings lead to the same lookup. This is the headline metric.

Ninety-five questions means each one moves the score by about 1.05 points. For that reason, raw counts are given beside the percentages. The judge has its own wobble as well: when it re-graded 75 already accepted rows at temperature 0, two verdicts changed. Differences under about 3 points are therefore treated as noise.

One prompt line that mattered

A side experiment came first. Two copies of the model were trained on identical data (the same 379 examples, split and hyperparameters), differing in a single paragraph. Copy A saw a fixed system message before each example that described the task and the YAML layout in plain words. Copy B saw only the question and the answer.

Going in, the bet was against the system message. With assistant_only_loss=True, loss is computed solely on the tokens of the model's own answer, the signal that drives learning. The system message and the question never receive a loss of their own, so it seemed they couldn't teach anything.

That bet lost. Copy A came out ahead on both metrics, by five questions each.

The result justified keeping the message, yet it shouldn't be oversold. Gains of 5.3 points (target correctness) and 5.2 points (exact match) clear the 3-point judge wobble. Other noise remains, though: only 95 test questions, and one training run per copy. Retraining the system-message setup from scratch later, for a different benchmark, gave 81/95 where the first run gave 83/95. A two-point swing from nothing but a new run means the true size of the gain across runs stays unknown.

Part of the why is clear. No loss lands directly on system message tokens, but that doesn't make them invisible to training. Gradients from the answer tokens flow back through the attention layers that read the system message, so the model can learn to use that context, and isn't limited to relying on whatever the base model already knew. Past that point it's educated guessing. One reasonable theory: spelling out the task and the schema gives the base model's next-token predictions a head start even before training, leaving fewer gaps to close. The released model keeps the system message.

A practical reading of the result: the instruction text is part of the model's input at inference time too, so the same short message must be present when the model is used. Dropping it at deployment would put the model into a situation it never saw during training.

The benchmark against bigger models

Once training finished, the model faced three reference points on the identical 95 questions: its own untouched base model, Llama-3.1-8B-Instruct served through Groq, and the GPT-4o teacher whose outputs made the labels.

Reading the first two columns side by side shows what fine-tuning contributes at a fixed prompt. Reading the tuned model against the other two shows how a small specialist fares against bigger models that each received a full prompt. The two prompts really are different things. The short one is the system message the small model trained and was tested on, listing the task and the three field names and nothing more. The long five-shot prompt is the one that produced the training labels. It brings five worked examples plus formatting rules the short version leaves out, like the literal string "Lookup" and the requirement that the output be a flat mapping instead of a list. Giving the larger models the long prompt was intentional. The aim was to see their performance with proper instructions, not to check whether they could guess an unstated format. With the short prompt, both landed near zero, and the cause wasn't weak reasoning but the missing format rules. The long prompt repaired it for both.

On accuracy, the small model falls behind Llama-3.1-8B, and by a larger gap behind GPT-4o. On speed it trails GPT-4o too, even with no network round trip. The reason is that GPT-4o sits on a serving stack tuned for fast, large-scale traffic, whereas the small model handles one call after another on a shared Colab GPU, where batching and weight caching are out of anyone's hands.

Llama-3.1-8B holds about 30 times more parameters. OpenAI keeps GPT-4o's size private, so no ratio is offered, but no one calls it small. Neither bigger model was beaten here. The benchmark does settle a different question: can a model this size handle this one narrow job? It can, at 81 out of 95. If the accuracy table is the only criterion, though, the small model doesn't win.

What the benchmark couldn't capture

The real case for this model doesn't come from that table. It comes from three things the table never measured.

Cost. Measured, not guessed. Token counts came from the actual prompts and completions in the GPT-4o runs, counted with tiktoken (the tokenizer behind OpenAI's models) and priced at OpenAI's published rates. A typical GPT-4o call consumed 606.6 prompt tokens and 39.4 completion tokens, about $0.0019 each, or near $1.91 for every 1,000 calls. The tuned model carries no per-call charge. Its hardware and power draw still cost something, so free isn't quite right, but there's no invoice per request and no rate limit.

Privacy. The two local inference functions make no network request of any kind. The question goes into the tokenizer, then the model, and stays on the device. This is a property of the code and not a benchmark number, so reading it is enough to verify. One caveat applies to the hosted demo: it runs on Hugging Face's servers, so text typed there leaves the visitor's machine. The privacy claim covers self-hosted runs only.

Offline operation. Checked twice, not assumed. First, HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 were set. Those flags make the Hugging Face libraries raise an error on any network attempt, instead of silently using a cache. Second, Wi-Fi was turned off entirely and the demo's own load-and-generate code ran with both flags still set. A brand-new question, absent from training and test data, still produced a correct decomposition with no connection.

No accuracy table shows these three traits. They're the real reasons for running a small model locally rather than calling a giant one over the wire, and the case reads as more honest with the unflattering accuracy numbers displayed right next to it.

What's available

The model, the dataset and a working demo are public on Hugging Face:

• the fine-tuned model
• the decomposition dataset
• a live demo space

The demo uses a shared free GPU pool, so the first request after idle time is slow while the weights load. It ships with two built-in questions, a bridge type and a comparison type, which lets visitors see both decomposition shapes without typing anything.

Checks continued past those two samples. A truly new question, in neither training nor test data ("The author of The Hobbit was born in which country?"), got a clean, correct decomposition. Not every try was flawless: a question on where SpaceX is headquartered produced the right target entity but a thought line that didn't make sense. Right now the choice of lookup is steadier than the explanation attached to it, particularly once phrasing drifts away from what HotpotQA typically uses.

What it adds up to

Take one tightly defined step of a bigger reasoning chain. A model of 270 million parameters handles it correctly in 81 of 95 cases, trails much larger models in accuracy, and still runs on a laptop with no bill and no network. For setups where questions can't leave the device, and where a wrong first lookup can be caught or retried later in the pipeline, Hop-1 is worth a trial. It doesn't demonstrate that a model this small can carry a whole reasoning chain by itself, with no checker behind it.