Why Indian Languages Cost More Tokens in LLMs and How Tokenizers Get Fixed

Say the same short sentence in five languages and count what an LLM has to chew through. The results aren't even close, and they have little to do with how hard a language is.

One sentence, five price tags

The measurements for a single short sentence:

Language	Characters	Bytes	Words	Tokens
English	18	18	4	5
Hindi	12	30	3	13
Telugu	17	45	3	20
Kannada	16	44	2	22
Tamil	30	84	3	34

The meaning is identical. The last column is the one that matters. Tokens are the small chunks a model actually reads, which might be a whole word, a fragment or just a few characters. English needed roughly one token per word. Tamil needed close to eleven.

This has nothing to do with Tamil being harder to understand. The gap comes from a choice made long before any prompt was typed: how the model's tokenizer got built. And the consequences are real. Conversations in Indian languages fill the context window sooner, cost more to run and come back slower, even when they say exactly what an English speaker says more cheaply.

What follows covers why the gap exists, what a small experiment shows about narrowing it, and what it takes to repair a tokenizer on a model that's already trained.

What a tokenizer does

Before a model processes a sentence, a tokenizer cuts it into tokens, the basic building blocks, a bit like Lego bricks. Everything after that, including the model's memory of the chat, the compute spent and the reply speed, gets counted in tokens and not words or letters. The table above is therefore a direct reading of how efficiently each language was chopped up.

Nobody designs that chopping by hand. It's learned from a huge pile of text through byte-level BPE, short for Byte Pair Encoding. The recipe is simple: start from raw bytes, find the pair that occurs next to each other most often in the training text, and merge it into a new token. Repeat. The first steps look like this:

t h  →  th
th e →  the

After thousands of rounds the result is a vocabulary in which every entry earned its place by being frequent. Pieces such as "the", "-ing" and "-tion" become single cheap tokens in an English-heavy corpus because they appear all the time.

Why bytes are a red herring

English letters take one byte each. Most Indic scripts, such as Devanagari, Telugu, Tamil and Kannada, take three bytes per character because their character sets are far larger. It's tempting to conclude that three bytes means three tokens, but that's wrong. A byte-level tokenizer can merge several bytes into a single token, so a three-byte character doesn't automatically cost three. What it costs depends on whether the tokenizer learned a good merge for that byte sequence, which in turn depends on how much text in that language it saw.

So the real explanation is training data and not script complexity. Train on 90% English and 10% everything else, and English patterns get thousands of opportunities to earn efficient one-token merges. Indic patterns get far fewer, so they stay in small, expensive pieces. The algorithm isn't biased in any moral sense. Frequency is simply the only thing BPE optimizes.

An experiment: does rebalancing the data help?

To test this, three small tokenizers were trained from scratch, with only the language mix changed between them:

• Tokenizer A: 90% English and 10% Indic languages, which mirrors how many real tokenizers get built since most web text is English
• Tokenizer B: an even 50/50 split between English and Indic text
• Tokenizer C: equal shares of English, Hindi, Telugu, Tamil and Kannada

One caveat applies. The corpus was small and self-assembled so that a single variable could be isolated. It isn't production scale, so the direction and relative size of the shifts matter more than the absolute figures.

Tokens per word on held-out test sentences:

Language	A (90% English)	B (50/50)	C (balanced)
English	3.05	3.08	3.60
Hindi	5.34	4.32	4.34
Telugu	7.79	6.36	6.36
Tamil	10.75	8.47	7.84
Kannada	8.86	7.34	7.07

Going from A to B, which gives Indic languages a fairer share, cut Hindi's cost by about 19%, Telugu's by about 18% and Tamil's by about 21%, while English hardly moved. For a while, rebalancing delivers a very good return.

Pushing on to the fully balanced mix gave diminishing returns for the Indic languages and took a bite out of English (3.08 up to 3.60). So more balance isn't always better. There's a sweet spot, not a straight line.

Would a bigger vocabulary fix it?

The same balanced corpus was used to sweep the vocabulary size. Tokens per word:

Language	300	800	1,500	3,000
English	5.28	3.70	3.70	3.70
Hindi	6.75	3.55	3.55	3.55
Telugu	9.64	6.42	6.42	6.42
Tamil	10.72	7.78	7.78	7.78
Kannada	10.17	7.17	7.17	7.17

Going from 300 to 800 helped every language a lot. After that, nothing changed, because the tiny corpus had run out of frequent patterns worth promoting to new tokens. At production scale, a curve with the same shape should be expected for a related reason: each added slot captures rarer and rarer patterns, so the gains shrink. Either way, a bigger vocabulary isn't automatically better. It enlarges the embedding table and the output layer, which makes it a real cost and not a free upgrade.

Neither lever, whether rebalancing or growing the vocabulary, works as a clean fix by itself. That raises the harder question of what to do with an already trained model that has a poor, English-heavy tokenizer, when starting over isn't an option.

Why the tokenizer can't simply be swapped

Replacing the old tokenizer with a better multilingual one is the obvious idea, and it fails because the tokenizer is wired deeply into the model.

Training gives every token its own embedding, a vector standing for what that token means, and the remaining layers operate on those vectors. Suppose ID 1234 maps to the embedding for "hello". If a new tokenizer assigns 1234 to a Hindi greeting instead, the embedding in that slot still means "hello". Nothing tells the transformer the meaning changed. The output turns incoherent, because the model's learned knowledge is still attached to the old IDs, and that connection has quietly snapped.

How teams retrofit languages: expand the vocabulary, freeze the weights

The technique used across the industry is vocabulary expansion, and it's done carefully so as to avoid exactly that breakage. In outline:

1. Leave the existing vocabulary alone. No current token ID is reassigned, since reassignment is what causes the mismatch.
2. Append new tokens for the target languages. A fresh set of merges, often trained on a multilingual or language-specific corpus, is added as new IDs after the existing ones, so nothing gets replaced. A vocabulary of 32K might grow past 100K.
3. Initialize the new embeddings. Random values work, though averaging related embeddings gives a warmer start, because a purely random vector tells a brand-new token nothing.
4. Lock the original weights and train just the new embeddings at first. This step carries the whole approach. Everything the model already knows about English, reasoning and code stays locked. Only the new embedding rows, and often the new rows of the output layer, can change. The model learns what the new tokens mean without risking catastrophic forgetting, the well-known failure where new learning overwrites old knowledge.
5. Continue pretraining on text in the new languages with the bulk of the model still locked, until the new embeddings carry enough meaning for the frozen layers to use them.
6. Unfreeze gradually. Sometimes the top layers go first and sometimes everything, and training continues so the whole network can adapt to the larger vocabulary.

Think of it as building additional rooms onto a house instead of demolishing and rebuilding it. Existing knowledge stays untouched through the early stages while the new rooms get furnished separately, and only afterward does the whole house rearrange itself around them.

This also shows why vocabulary changes aren't purely an input-side matter. The model must read the new tokens, which is the job of the input embeddings, and also produce them in replies. That falls to the output layer, sometimes called the LM head, which scores every entry in the vocabulary as a possible next token. Training must cover both ends, and whether the two layers share weights varies by architecture.

A note on precision: the exact production recipes, meaning freezing schedules, chosen vocabulary sizes and the duration of each phase, are proprietary and specific to each team, so none of them should be pinned on a particular company without a public source. The staged pattern of expanding, warming up and gradually unfreezing is the general, documented approach, while the specific numbers aren't public.

What this doesn't prove

Token efficiency isn't the same as model quality. A low tokens-per-word figure doesn't show that a model understands the language, thinks clearly in it or translates it well. Cheap tokenization can go with weak modeling, and costly tokenization can go with strong understanding. All of this concerns the economics layer, meaning cost, latency and context usage, and not comprehension.

The chain, end to end

Different scripts get different byte representations. BPE learns merges according to frequency in the training data. Underrepresented languages end up with fewer efficient merges. Those languages therefore need more tokens to say the same thing, which means less usable context, slower replies and higher bills for identical conversations.

Rebalancing the data and enlarging the vocabulary both help, but both plateau and both have real costs. Vocabulary expansion with frozen weights, handled carefully to avoid catastrophic forgetting, is more involved than either lever alone. That's exactly why teams reach for it instead of retraining a bigger, more balanced tokenizer from scratch.

For multilingual LLMs, the tokenizer isn't a preprocessing afterthought. It's part of the economics of language, and fixing it properly after training takes real engineering, not just better data. Keep in mind that the figures above come from a toy BPE setup, built to vary one thing at a time, and say nothing about how any shipping tokenizer performs.