Tokenization Is a System Design Decision, Not a Preprocessing Step

Four symptoms with one cause

Sooner or later, most people building on LLMs run into one of these:

• A model advertised with a 128k context window runs out of room far earlier than expected on non-English or code-heavy input.
• One feature costs noticeably more per request for users writing Japanese, Hindi or Arabic than for English writers.
• The model confidently fails to count letters or reverse a word, however the prompt is phrased.
• Two teams report different token counts for text that looks identical, and nobody can say why.

Rewording the prompt won't solve any of these, and the model isn't simply being stupid. The culprit is tokenization, the pipeline stage engineers tend to know least. It lives between your code and the model, does its job unseen, and only gets noticed when it fails. What follows explains the mechanism, its place in a real system, the damage it can cause and a checklist to run before blaming the model.

What the model actually receives

A transformer can't process text directly, so the first thing that happens is conversion into integers. That conversion is tokenization, and it relies on a vocabulary that was fixed beforehand. Nothing in your prompt can alter it while the model runs.

Picture a fixed set of Lego pieces. Common English words such as "the," "is" and "running" are usually one piece. Rarer words are split, so "tokenization" might become "token" plus "ization." Text the tokenizer saw little of, like a rare language, an unusual Unicode symbol or an obscure product name, breaks into more and smaller pieces.

Your raw characters never reach the model. What arrives is whatever segmentation the tokenizer chose, and billing, context use and certain reasoning errors all trace back to where it cut. The word "strawberry" is a good example: the model sees a few tokens and not ten letters.

How the vocabulary gets built

Most vocabularies come from Byte Pair Encoding (BPE). Its origin is in compression, not language: Philip Gage described it in 1994 as a way to shrink data by substituting a fresh symbol for the pair of symbols that appears next to each other most often.

Sennrich, Haddow and Birch brought it to neural machine translation in 2016 with the paper "Neural Machine Translation of Rare Words with Subword Units." Word-level vocabularies struggle with rare words, typos and word forms, and their answer was an open vocabulary where rare words are assembled out of frequent, smaller subword pieces.

The GPT-2 work from OpenAI (Radford et al., 2019, "Language Models are Unsupervised Multitask Learners") pushed the idea to the byte level. Merging begins from raw UTF-8 bytes instead of characters, which gives a starting alphabet of exactly 256 symbols. That single change lets a modern tokenizer represent any string in any language, including malformed text and emoji, without hitting an unknown-token wall. Nearly every major model family today, GPT and Llama among them, uses a variant of it.

Training happens once, offline, in these steps:

1. Start with base units, either the 256 bytes or characters in the original formulation.
2. Scan a large corpus and count how often each adjacent pair of symbols occurs.
3. Take the pair that occurs most often, fuse it into one new symbol and put that symbol in the vocabulary.
4. Replace every occurrence of that pair in the corpus with the new symbol.
5. Repeat steps 2 to 4 until the vocabulary hits a target size, commonly somewhere between tens of thousands and a couple hundred thousand tokens.

Once training is over, encoding is fast and fully deterministic. New text gets the learned merges applied in the same order they were learned, joining pieces greedily until nothing more can merge. No understanding is involved. It's lookup and merge, and the same input always produces the same tokens.

SentencePiece, a separate library from Google (2018), takes a different route that's worth knowing. It trains directly on raw text and treats whitespace as an ordinary symbol, not a special delimiter. That matters for Chinese, Japanese and Thai, which don't separate words with spaces, where a whitespace-based pretokenizer falls apart and SentencePiece copes uniformly.

Where it sits in a request

Tokenization runs before the transformer. The path of a real request looks like this:

User Text
   ↓
Pretokenizer (split on whitespace/punctuation rules, handle Unicode)
   ↓
BPE Encoder (apply learned merges → token IDs)
   ↓
Token IDs (integers)
   ↓
Embedding Lookup Table
   ↓
Transformer (attention + feedforward layers)
   ↓
Output Token IDs
   ↓
Decoder (token IDs → text)
   ↓
Response Text

Two components are easy to forget. First, the embedding table is indexed by token ID, so vocabulary size sets the size of the embedding and output-projection layers. A bigger vocabulary spends more parameters on mapping tokens to vectors before any reasoning layer runs. Second, the decoder is the tokenizer in reverse. Streaming a response token by token and turning it back into readable text relies on the same fixed vocabulary used for encoding.

Seeing it with tiktoken

Rather than taking this on faith, you can look at it with tiktoken, OpenAI's tokenizer library:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

samples = {
    "english": "The invoice was approved yesterday.",
    "japanese": "請求書は昨日承認されました。",
}

for label, text in samples.items():
    tokens = enc.encode(text)
    print(f"{label}: {len(tokens)} tokens -> {tokens}")
    print([enc.decode([t]) for t in tokens])

Two things to check when you run a comparison like this. First, tokens per character or word vary widely by language. The same information can cost noticeably more tokens depending on how well a language was represented when the tokenizer was trained. OpenAI's own tokenizer documentation shows it: one short phrase breaks into more pieces under older, English-centric encodings than under newer ones, and non-Latin scripts generally use more tokens per character than English prose. Second, token boundaries rarely line up with characters or even syllables. That's exactly why letter counting and character reversal are hard for LLMs by default, since the working representation doesn't expose individual characters as a first-class unit.

A multilingual budget example

Imagine a support agent serving English, Japanese and Arabic speakers under a hard token budget per request. Capping input and output tokens per call to bound cost and latency is a normal production practice. If you size that budget by testing in English alone, you'll systematically shortchange everyone else. The same conversation history and system prompt eat more of the budget in Japanese or Arabic and leave less room for the answer.

Nothing about this is exotic. BPE tokenizers trained mostly on English behave this way out of the box, and the result is a silent loss of quality, through cut-off context, for the international users the product is meant to reach. A cleverer prompt doesn't fix it. What does help is counting real tokens for each language while testing, then choosing budgets, truncation or summarization rules separately for each one.

A cheap experiment

Before trusting assumptions about tokenizer behavior, try this. Take 20 to 30 sentences with equivalent content, professionally translated into English, Japanese, Arabic and Hindi (support transcripts work well). Encode each with one tokenizer, such as o200k_base through tiktoken, and measure tokens per sentence and tokens per character for each language.

The hypothesis to check (it isn't an established finding) is that Japanese, Arabic and Hindi come out with a distinctly larger ratio of tokens to characters than English does. That would mean the same context window and the same per-request budget leave those users less effective room for the same content. The exact ratio depends on the tokenizer version and the languages, so measure it on your own data.

Failure modes you'll meet in production

Tokenization produces a specific, recurring set of bugs.

• Character-level task failures. Counting letters, reversing strings, spelling and some multi-digit arithmetic fail because the model reasons over subword tokens and not characters. The symptom is a model that's confidently wrong on something trivial for a human. Better prompting alone won't fix it. Asking the model to spell the word out with separators, which forces character-level tokens, or handing the task to code execution, works better.
• Tokenizer and model mismatch. Using another model's tokenizer to estimate cost or context usage gives systematically wrong numbers. You'll see local token counts that disagree with the API bill. Use the encoding documented for the model you call.
• Overflow nobody notices. A prompt that fits easily in English can exceed the limit once translated, and the same goes for content full of code, JSON or odd symbols. The symptom is truncated context and worse answers, with no explicit error in some pipelines. Measure real token counts on real production content.
• Whitespace and formatting sensitivity. Leading spaces, repeated whitespace and inconsistent formatting can shift token boundaries and counts, sometimes by a lot, so two prompts that look the same to a person differ in tokens and cost.
• Vocabulary mismatch after fine-tuning or model swaps. Moving to a different model family with a different tokenizer, without rechecking prompt budgets or few-shot lengths, can quietly break assumptions your system relied on. Something that worked fine degrades after a supposed drop-in swap.

What to do in production

• Cost: tokens are the real billing unit, so multilingual products should budget per language and not assume one global average.
• Context planning: size RAG chunks, conversation-history truncation and few-shot examples in tokens of the specific tokenizer you use, and never in words or characters.
• Speed: tokenization itself is fast and rarely the bottleneck, but estimating prompt length with the wrong encoding before a call can lead to needless retries or truncation, and both add delay.
• Observability: log actual input and output token counts per request, split by language if you serve several locales. It's cheap to add and catches cost and context problems before users report them.
• Consistency: use the same tokenizer version for local testing, cost estimation and the production API call. A mismatched encoding gives confidently wrong numbers.

When you can ignore all this

For a purely English, short-form, low-volume application, such as an internal tool, a prototype or a low-traffic chatbot, tokenizer-level concerns like multilingual cost gaps aren't worth engineering time. The library defaults are fine and your effort is better spent elsewhere.

Likewise, if the product never needs character-level work like spelling, letter counting or exact string reversal, there's nothing to design around. It only matters when the use case really asks the model for those tasks, and then the usual fix is delegating to a code-execution step instead of fighting the tokenizer through prompts.

The short version

Models never see characters or words, only tokens from a fixed vocabulary built ahead of time, typically with byte-level BPE. BPE began as a 1994 compression algorithm, was brought into NLP in 2016 and extended to bytes by the GPT-2 team in 2019, which is why modern tokenizers essentially never emit unknown tokens. Token counts, and not word or character counts, set your real cost and context use. The same information can cost more tokens in some languages, which quietly disadvantages non-English users unless you budget per language. Character counting, spelling and reversal failures come from the tokenizer's representation and aren't a reasoning deficit, and mismatched tokenizers cause avoidable production bugs. Measure tokenization and keep watching it as part of the design, instead of forgetting it after the first integration.

References:

• Gage (1994), "A New Algorithm for Data Compression," The C Users Journal, the original BPE compression algorithm.
• Sennrich, Haddow and Birch (2016), "Neural Machine Translation of Rare Words with Subword Units," ACL 2016, DOI 10.18653/v1/P16-1162.
• Radford et al. (2019), "Language Models are Unsupervised Multitask Learners," the GPT-2 paper introducing byte-level BPE.
• Kudo and Richardson (2018), the SentencePiece paper from Google, a subword tokenizer and detokenizer that doesn't depend on language.
• The OpenAI Cookbook guide to counting tokens with tiktoken, and the tiktoken documentation for the cl100k_base and o200k_base encodings.