Fragile Tokens and Silent Substitution: A Geometric Account of Some Hallucinations
Some hallucinations come from damaged or fragile tokens that get swapped for near neighbors before reasoning starts. Examples, per-model rates and a simple test.
This article is published in English.
Sources: Hallucinations Related to Glitch and Fragile Tokens
Users of open-source models, Qwen3.8 among them, report fewer hallucinations, better instruction following and steadier task reliability. No published metric appears to isolate hallucination rates across model generations. One thing stands out in most discussions: they rarely propose any geometric mechanism. Hallucination tends to be treated as a single diffuse property that improves with more training, not as a set of separate failure modes that can be measured and attributed one by one.
At least one such failure mode can be found and quantified ahead of time.
A hallucination that can be checked directly
OLMo-2-7B was given a four-item list and asked for the second item, answering with the item alone. The list read lantern, a Russian word meaning "which", harbor and compass. The model replied with bort, a Russian word meaning roughly "side" or "board".
Nothing in the prompt contains that word, which is simply a Russian term with no connection to the list. It was delivered without hesitation to a question that needs only basic counting. Qwen3.8-27B got the same task with a different Russian word in second position and answered "oxtend", a word that exists in no language, apart from what seems to be a drug in clinical trials. That probably isn't what the model meant, but in an agentic task it could be disastrous.
This is hallucination in its strictest sense: a confident, specific and false statement about information the model could have verified from its own input. And unlike most hallucinations, the origin of the error can be identified.
The pattern isn't limited to toy prompts. In a spelling-correction request about filing a form before Friday, the form's name had been replaced by one rare Chinese character. Qwen2.5-7B answered that the correct spelling was waiver and not :numel. Both halves were made up: ":numel" never appeared in the input, and "waiver" appeared nowhere either. In another test the same model received a Hebrew verb meaning "to indicate" and was asked its meaning, and it explained a Chinese word for "pointer" instead. Qwen3-1.7B was shown one unusual Korean syllable, copied it back correctly on screen, then called it a Chinese character meaning full or abundant.
In none of these cases does the model hesitate, hedge or ask for clarification. The answers are fluent and elaborate, yet they never address the question that was posed.
What the cases share
The usual picture of hallucination has a model reaching past the edge of its knowledge and inventing something to fill the gap. That does happen, but it isn't what happens here.
In each example the model misread its input and then reasoned correctly from the misreading. No fabrication and no knowledge limit were involved. The model was asked about one item, received a different one and responded sensibly to that. From outside, the behavior can't be told apart from invention. Inside, the model registers no error, which explains why its output carries no uncertainty.
The proposed explanation is that a small slice of every model's vocabulary is made of damaged tokens. Some can't even be reproduced alone, like the glitch tokens found after the SolidGoldMagikarp discovery in GPT-2. Others look fine by themselves but break inside normal sentences, and the recent measurements concentrate on this second kind. Faced with a fragile or damaged token, the model doesn't just produce noise. It slides toward a completion that is confident and low in variance: a near neighbor, a shortened form or a translation. A Chinese token for "query" comes back as English "QUERY". Italian "realtà" returns as French "réalité". A truncated Russian form of "chairman" becomes the Polish "Przewodniczacy", which happens to be a correct translation.
Near-neighbor substitution as the mechanism
As a model moves through a completion, it may hit a fragile or damaged token along its embedding trajectory. The readout resolves it to a healthy neighbor, and every later transformation proceeds as if the user had written that neighbor. The substitution isn't a mistake the model recovers from. It becomes the input the model reasons about.
Each link in the chain deserves spelling out, because each is something the model does correctly:
- A damaged token influences the completion process.
- A substituted token enters the sequence, indistinguishable from anything the user wrote.
- Every later step is computed from that substitute.
- Nothing gets flagged, since from the model's viewpoint nothing went wrong.
- The completion reads sensibly while the task has silently degraded, and the model stays confident because no uncertainty ever arose.
A hallucination is convincing because it is fluent, confident and specific, and in this case all three properties fall straight out of the embedding geometry.
For the list question, this yields an answer like bort, which is ultimately a curiosity. The consequences for agentic systems are bigger. Qwen3-32B was given a research task and a set of research tools. It started a search, but the query dropped or altered tokens, so the results were irrelevant, and the agent then cited those irrelevant references with confidence. Across fifty trials, the tokens at issue appeared in neither the search nor the answer, while stable tokens with no fragile neighbors showed up in both about 96 percent of the time.
Everyday words are affected too
If the effect were limited to rare scraped strings, its practical weight would be small. It extends to common words.
Take a model's known-bad tokens and score every other token by how close it sits to that region of the output embedding. The score predicts which healthy-looking tokens will fail in ordinary sentences, with an AUC between 0.72 and 0.83 on six of the eight models examined. A random reference region of the same size gives 0.41 to 0.55.
Fragile tokens needn't appear in the prompt at all. Close neighbors of them are enough. The sharpest example found is the English word "according". On OLMo-2-7B it is reproduced alone with probability 0.999, yet it breaks in 44 of 48 ordinary sentences. Given a sentence about filing the "according" form before Friday and asked to fix the spelling, the model returned a version with "correct" in its place, deleting a common English word and smoothing over the hole.
How much of the vocabulary is at risk
The measure is the share of otherwise clean tokens that fail in at least one of ten contexts:
Measured this way, Qwen3.8-Flash-Next sits at 0.4 percent and Qwen3-32B at 0.9. Qwen3.8-27B reaches 1.5, Qwen3-1.7B 2.4 and OLMo-2-7B 2.7. The older and larger models are far worse, with Qwen2.5-7B at 7.0, Qwen2.5-72B at 10.5 and Qwen3-235B at 10.9 percent.
That's a spread of about twenty-five times. One model has about 1 in 250 tokens exposed, while another has 1 in 10.
Vocabulary size isn't the explanation. Take only the token IDs present in both Qwen3-32B and Qwen3-235B and judged clean by each. On that identical set, the 235B model fails 4.3 percent of the time and the 32B model 0.8 percent, a fivefold gap.
A map of the embedding space makes the contrast visible, with each dot a token and tokens the model treats alike placed close together. Affected tokens are marked in red. The cleanest model in the set, Qwen3.8-Flash-Next, has fifteen of them, while Qwen3-235B has 682.
The hypothesis
This family of hallucinations doesn't stem from a knowledge gap. A token falls inside a damaged region of the embedding space and gets replaced before any reasoning begins, so the model answers a question nobody asked. The error rate should track how much of the vocabulary lies in that region, and the numbers above show that share varying by more than ten times between current models.
If that holds, the reduced hallucination reported for newer models may reflect a cleaner vocabulary and a more sound embedding geometry, not better knowledge. Qwen3.8 holds the two lowest positions in the list. Published hallucination metrics that isolate these generations haven't turned up, so the idea remains unverified.
A separate measurement pointing the same way
PhantomBench, created by Haeji Jung and Hila Gonen, gathers over 60,000 invented terms and entities and tests 21 models with them. Average failure-to-abstain rates reach as high as 86.7 percent, frontier models included.
What it studies, behavior on entities that don't exist, is functionally close to hitting a damaged token. The authors report that responses to non-existent concepts follow responses to rare ones, and damaged tokens belong to the rare tail. They also note that the largest model in each family shows a sudden rise in hallucination, with Qwen3-32B and Llama 3-70B doing worse than smaller siblings. The token measurements likewise show the biggest models as the most affected. Still, the sets of models in the two studies overlap too little to support a correlation.
An experiment that would settle it
Part of this is speculation, though the token measurements make further testing worthwhile. The test is straightforward. Run one model through a hallucination benchmark two times: first with the standard prompts, then with each affected token swapped for a stable synonym. A lower rate in the second run shows how big the effect is. An unchanged rate would mean the proposed link doesn't exist.
Reference: the PhantomBench paper by Jung and Gonen, arXiv:2606.11105.