BPE Tokenizer From Scratch: Shorter Sequences for a Small GPT Context Window
Write a Byte Pair Encoding tokenizer in Python, train it on WikiText-2, save merges as JSON, and see how subword tokenization stretches a GPT context window.
This article is published in English.
Sources: Tutorial 16: Beyond Character Tokens, Building a BPE Tokenizer
A character-level tokenizer is the easiest way to get a small GPT working: the vocabulary is tiny, nothing is ever unknown, and every step is transparent. The price is sequence length, since every letter costs a position in the context window. This guide replaces it with Byte Pair Encoding (BPE), written from scratch in plain Python. You will train it on WikiText-2, persist the learned vocabulary and merges as JSON, fingerprint it, re-encode the dataset, and see exactly what changes when you retrain the model on subword tokens.
The code assumes an existing small PyTorch GPT and cleaned WikiText-2 files. If training still misbehaves, stabilize it first; see tests that isolate each failure in a small PyTorch GPT.
Why character tokens stop being enough
A character tokenizer gives every character, including spaces, its own token. Take this short phrase:
artificial intelligence
It turns into one token per character:
['a', 'r', 't', 'i', 'f', 'i', 'c', 'i', 'a', 'l',
' ', 'i', 'n', 't', 'e', 'l', 'l', 'i', 'g', 'e',
'n', 'c', 'e']
That is 23 tokens for two words. A subword tokenizer could represent the same phrase with far fewer pieces, for example:
['art', 'ificial', ' intelligence']
Three tokens instead of 23. Shorter sequences mean more text per context window, fewer positions to train on and fewer decoding steps at generation time. The model also receives meaningful chunks instead of isolated letters, frequent fragments become reusable units, and rare words can still be spelled out from smaller pieces. The hard part is discovering good subword units automatically, which is precisely the job BPE does.
Characters, words and subwords compared
There are three broad ways to split text, and each makes a different trade-off.
Character tokenization
Every character is a token:
"learning"
→ ["l", "e", "a", "r", "n", "i", "n", "g"]
It offers a small vocabulary, no unknown tokens and a trivial implementation. In exchange you get long sequences, slow generation, and a model that must assemble words from single letters.
Word tokenization
Every word is a token:
"machine learning works"
→ ["machine", "learning", "works"]
Sequences are short and readable, but the vocabulary becomes enormous, unseen words need an unknown token, and related forms such as "learn", "learning" and "learned" can end up as three unrelated IDs with nothing shared between them. Names, misspellings and rare words are handled badly.
Subword tokenization
Words are split into frequent fragments:
"unbelievable"
→ ["un", "believ", "able"]
This gives shorter sequences than characters and a smaller vocabulary than words. Frequent words can still become single tokens, rare words decompose into known pieces, and fragments such as prefixes and suffixes get reused across many words. The only real cost is that training and implementing the tokenizer is more involved.
You can think of the three approaches as points on a spectrum:
Characters
← flexibility
Subwords
← balance
Words
← compactness
How BPE learns its merges
BPE starts with the smallest units, here individual characters, and repeatedly fuses the most common pair of neighbors into a new token. Consider a tiny corpus of three words:
low
lower
lowest
Initially each word is just its characters:
l o w
l o w e r
l o w e s t
Next, list the pairs of adjacent symbols that occur:
(l, o)
(o, w)
(w, e)
(e, r)
(e, s)
(s, t)
One pair stands out:
(l, o)
It occurs in all three words, so it is merged into a single symbol:
(l, o) → lo
After replacing it everywhere, the corpus looks like this:
lo w
lo w e r
lo w e s t
Counting again, the most frequent pair is now:
(lo, w)
It also appears three times, so it gets merged too:
(lo, w) → low
And the corpus becomes:
low
low e r
low e s t
After enough rounds, the most frequent sequences become tokens of their own.
What the training loop produces
The procedure you are about to implement follows these steps:
Training text
→ initialize character tokens
→ count adjacent token pairs
→ select most frequent pair
→ merge the selected pair
→ add merged token to vocabulary
→ repeat until target vocabulary size
It produces two artifacts, and you need both to use the tokenizer later.
The vocabulary
The vocabulary maps each token string to an integer ID, for example:
"a" → 0
"b" → 1
"th" → 98
"ing" → 127
" the" → 204
The merge rules
The merge rules are the ordered list of pairs that were fused during training:
("t", "h") → "th"
("i", "n") → "in"
("in", "g") → "ing"
(" ", "the") → " the"
Order is essential. A later rule such as ("in", "g") can only fire if the earlier rule that created "in" has already been applied. Applying the same rules in a different order produces a different segmentation.
A first implementation on a toy corpus
Before touching WikiText-2, work through the algorithm on a string you can inspect by eye:
text = "low lower lowest"
Splitting it into characters gives the starting token list:
tokens = list(text)
print(tokens)
Printing it shows every character, including the spaces:
['l', 'o', 'w', ' ', 'l', 'o', 'w', 'e', 'r',
' ', 'l', 'o', 'w', 'e', 's', 't']
To find candidates for merging, count every pair of neighbors. Zipping the list with itself shifted by one position yields each adjacent pair, and Counter tallies them:
from collections import Counter
def count_pairs(tokens):
return Counter(
zip(tokens[:-1], tokens[1:])
)
Inspect the most common pairs:
pair_counts = count_pairs(tokens)
print(pair_counts.most_common(10))
The output will look roughly like this:
[(('l', 'o'), 3),
(('o', 'w'), 3),
(('w', 'e'), 2),
...]
Note that (' ', 'l') also appears in the counts. Because spaces are ordinary characters in this scheme, BPE can learn tokens that include a leading space, which is how tokens such as " the" arise later.
Merging a single pair
The core operation walks the token list from left to right. Whenever the current token and the next one match the chosen pair, it emits the merged token and skips ahead two positions; otherwise it copies the current token and advances by one. Overlaps resolve left to right: in a a a with pair (a, a), only the first two merge.
def merge_pair(
tokens,
pair,
merged_token,
):
output = []
index = 0
while index < len(tokens):
if (
index < len(tokens) - 1
and tokens[index] == pair[0]
and tokens[index + 1] == pair[1]
):
output.append(merged_token)
index += 2
else:
output.append(tokens[index])
index += 1
return output
Apply it to merge l and o:
tokens = merge_pair(
tokens=tokens,
pair=("l", "o"),
merged_token="lo",
)
print(tokens)
The result is shorter than the input:
['lo', 'w', ' ', 'lo', 'w', 'e', 'r',
' ', 'lo', 'w', 'e', 's', 't']
Every merge removes one token for each occurrence of the pair, which is exactly where the compression comes from.
Repeating until the vocabulary is full
A minimal trainer simply loops: count pairs, pick the most frequent one, merge it everywhere, record the new token and the rule, and stop once the vocabulary reaches the target size. It also stops early if no pairs remain, or if the best pair occurs fewer than twice, because at that point merging no longer creates anything reusable.
def train_bpe_on_tokens(
tokens,
target_vocab_size,
):
vocabulary = set(tokens)
merges = []
while len(vocabulary) < target_vocab_size:
pair_counts = count_pairs(tokens)
if not pair_counts:
break
best_pair, frequency = (
pair_counts.most_common(1)[0]
)
if frequency < 2:
break
merged_token = "".join(best_pair)
tokens = merge_pair(
tokens=tokens,
pair=best_pair,
merged_token=merged_token,
)
vocabulary.add(merged_token)
merges.append(best_pair)
print(
f"Merged {best_pair} "
f"into {merged_token!r}, "
f"frequency={frequency}"
)
return tokens, vocabulary, merges
Running it on the toy string with a small target is enough to watch merges happen one by one:
final_tokens, vocabulary, merges = (
train_bpe_on_tokens(
tokens=list(text),
target_vocab_size=20,
)
)
This version is good for building intuition, but every iteration rescans the entire Python list. On the full WikiText corpus that becomes very slow, so the practical version below trains on a configurable slice of WikiText-2 instead.
Designing a reusable tokenizer
The tokenizer class should train from raw text starting at characters, encode and decode, reject characters outside its vocabulary, save to and load from JSON, and compute a fingerprint. Around it, a script re-encodes every WikiText-2 split and reports how much shorter BPE sequences are.
One design decision is worth stating clearly. This tokenizer works on Unicode characters, not on raw bytes. Production tokenizers used by large models are usually byte-level, which means their base vocabulary is the 256 possible byte values and no input can ever be unrepresentable. The character-based version here follows the same algorithm and is easier to read, but it is not a drop-in equivalent of those tokenizers.
The class skeleton
The constructor takes the base characters and the ordered merge list. It builds the full token list by starting with the base characters and appending the result of each merge, skipping duplicates. From that list it derives stoi (string to ID) and itos (ID to string) lookups and records the vocabulary size. Because IDs come from list positions, the same base tokens and merges always yield the same IDs.
class BPETokenizer:
def __init__(
self,
base_tokens,
merges,
):
self.base_tokens = list(
base_tokens
)
self.merges = [
tuple(pair)
for pair in merges
]
self.tokens = list(
self.base_tokens
)
for left, right in self.merges:
merged = left + right
if merged not in self.tokens:
self.tokens.append(merged)
self.stoi = {
token: index
for index, token in enumerate(
self.tokens
)
}
self.itos = {
index: token
for token, index
in self.stoi.items()
}
self.vocab_size = len(
self.tokens
)
The resulting vocabulary is therefore made of two parts:
Base character tokens
+
learned merged tokens
Training on real text
Training becomes a class method that returns a ready-to-use tokenizer. It sorts the unique characters of the text to form the base vocabulary, refuses a target size smaller than that base set, and then runs the same count, pick and merge loop as before. Logging is throttled: it prints each of the first 20 merges and then every hundredth one, along with the pair frequency and the current sequence length, so you can watch compression progress without drowning in output.
@classmethod
def train(
cls,
text,
target_vocab_size,
minimum_pair_frequency=2,
verbose=True,
):
base_tokens = sorted(
set(text)
)
if target_vocab_size < len(base_tokens):
raise ValueError(
"target_vocab_size cannot be "
"smaller than the number of "
"unique characters"
)
working_tokens = list(text)
vocabulary = set(base_tokens)
merges = []
while len(vocabulary) < target_vocab_size:
pair_counts = Counter(
zip(
working_tokens[:-1],
working_tokens[1:],
)
)
if not pair_counts:
break
best_pair, frequency = (
pair_counts.most_common(1)[0]
)
if frequency < minimum_pair_frequency:
break
merged_token = "".join(
best_pair
)
working_tokens = merge_pair(
tokens=working_tokens,
pair=best_pair,
merged_token=merged_token,
)
merges.append(best_pair)
vocabulary.add(merged_token)
if (
verbose
and (
len(merges) <= 20
or len(merges) % 100 == 0
)
):
print(
f"Merge {len(merges):4d}: "
f"{best_pair!r} "
f"→ {merged_token!r}, "
f"frequency={frequency:,}, "
f"tokens={len(working_tokens):,}"
)
return cls(
base_tokens=base_tokens,
merges=merges,
)
Why there is a minimum pair frequency
If the most frequent remaining pair occurs only once, merging it creates a token that will never be reused, which just wastes a vocabulary slot. The loop therefore stops under this condition:
frequency < minimum_pair_frequency
For experiments like this one, a threshold of two is enough:
minimum_pair_frequency = 2
On a larger corpus you may want a higher threshold, so that the vocabulary is spent only on patterns that occur often enough for the model to learn good embeddings for them.
Encoding and decoding with the tokenizer
Encoding follows three steps:
- Break the text into base characters.
- Replay every merge rule in the order it was learned.
- Look up the ID of each resulting token.
The first half of that work lives in encode_tokens. It starts by collecting any characters that are missing from the vocabulary and raises a clear error if there are some, which is far easier to debug than a KeyError halfway through a dataset. Then it replays the merges.
def encode_tokens(self, text):
unknown_characters = sorted({
char
for char in text
if char not in self.stoi
})
if unknown_characters:
raise ValueError(
"Text contains characters missing "
"from the base vocabulary: "
f"{unknown_characters}"
)
tokens = list(text)
for pair in self.merges:
merged_token = (
pair[0] + pair[1]
)
tokens = merge_pair(
tokens=tokens,
pair=pair,
merged_token=merged_token,
)
return tokens
Turning the resulting token strings into IDs is a straightforward lookup:
def encode(self, text):
tokens = self.encode_tokens(text)
return [
self.stoi[token]
for token in tokens
]
Decoding is even simpler. Every token, merged or not, stores the exact text it covers, so decoding is just concatenation:
def decode(self, token_ids):
return "".join(
self.itos[int(token_id)]
for token_id in token_ids
)
That gives BPE a property worth testing explicitly, lossless round-tripping:
decode(encode(text)) = text
A quick check with a sample sentence:
sample = "The history of science."
encoded = tokenizer.encode(sample)
decoded = tokenizer.decode(encoded)
assert decoded == sample
Persisting and reloading the tokenizer
You do not need to store the vocabulary itself, because it can be rebuilt deterministically from the base tokens and the merge list. The save method writes those two pieces to JSON, along with a type label and the vocabulary size for convenience. Setting ensure_ascii=False keeps non-ASCII characters readable in the file, and the parent directory is created if it does not exist yet.
def save(self, path):
data = {
"type": "character_bpe",
"base_tokens": self.base_tokens,
"merges": [
[left, right]
for left, right in self.merges
],
"vocab_size": self.vocab_size,
}
path = Path(path)
path.parent.mkdir(
parents=True,
exist_ok=True,
)
with open(
path,
"w",
encoding="utf-8",
) as file:
json.dump(
data,
file,
ensure_ascii=False,
indent=2,
)
The result is a single file:
tokenizer/
└── bpe_tokenizer.json
Loading reverses the process by reading the JSON and passing both lists to the constructor:
@classmethod
def from_file(cls, path):
with open(
path,
"r",
encoding="utf-8",
) as file:
data = json.load(file)
return cls(
base_tokens=data[
"base_tokens"
],
merges=data["merges"],
)
Then confirm that a reloaded tokenizer still round-trips the sample text:
loaded_tokenizer = (
BPETokenizer.from_file(
"tokenizer/bpe_tokenizer.json"
)
)
assert (
loaded_tokenizer.decode(
loaded_tokenizer.encode(sample)
)
== sample
)
Training on a slice of WikiText-2
Because the simple trainer rescans the whole working sequence on every merge, training on all of WikiText-2 with a large vocabulary can take a long time. Start with a configurable prefix of the training text:
training_characters = 500_000
training_text = train_text[
:training_characters
]
A reasonable first vocabulary size is:
target_vocab_size = 500
In other words:
Initial character vocabulary
+
learned merged tokens
=
approximately 500 tokens
When everything works, try larger targets:
target_vocab_size = 1_000
target_vocab_size = 2_000
Expect training time to grow with both the size of the sample and the number of merges.
Taking base characters from every split
If the base vocabulary is built only from the training sample, the validation or test text may contain characters that never appeared in it, and encoding those splits will fail. The fix is to collect the base character set from all three splits, while still learning merge statistics only from training data:
all_text = (
train_text
+ val_text
+ test_text
)
base_tokens = sorted(
set(all_text)
)
This does not leak any validation language patterns into the merges. It only guarantees that every character in the dataset has an ID. To support it, the trainer gains an optional base_tokens argument that is used instead of the characters of the training text when provided:
@classmethod
def train(
cls,
text,
target_vocab_size,
minimum_pair_frequency=2,
base_tokens=None,
verbose=True,
):
if base_tokens is None:
base_tokens = sorted(
set(text)
)
else:
base_tokens = sorted(
set(base_tokens)
)
Measuring the compression you gained
To see what BPE buys you, compare token counts on the same passage. Take a sample:
sample_text = train_text[:10_000]
Its character token count is simply its length:
character_count = len(sample_text)
Its BPE token count is the length of the encoded ID list:
bpe_ids = tokenizer.encode(
sample_text
)
bpe_count = len(bpe_ids)
The ratio between them is the compression factor:
compression_factor = (
character_count / bpe_count
)
A result might look like this:
Character tokens: 10,000
BPE tokens: 4,250
Compression: 2.35×
With a compression factor of 2.35, a block of 128 BPE tokens covers roughly as much text as 301 character tokens would. Compression is not uniform, though: prose full of common words compresses well, while numbers, rare names and unusual punctuation compress poorly.
Looking at what the tokenizer learned
The learned tokens are everything in the vocabulary after the base characters. Sorting them by length and printing the longest ones, with their IDs, gives a quick sense of what the tokenizer considers frequent:
learned_tokens = (
tokenizer.tokens[
len(tokenizer.base_tokens):
]
)
longest_tokens = sorted(
learned_tokens,
key=len,
reverse=True,
)
for token in longest_tokens[:50]:
print(
repr(token),
tokenizer.stoi[token],
)
You will typically see tokens along these lines:
" the"
"ing"
"tion"
" of"
"ed"
" and"
The exact list depends on the training sample, preprocessing, target vocabulary size, minimum frequency and tie-breaking. On tie-breaking: Counter.most_common returns equally frequent pairs in the order they were first counted, so results are deterministic for a given input, but changing the sample can reshuffle which of several tied pairs wins.
Re-encoding the dataset
With a trained tokenizer, encode all three splits:
train_bpe_ids = tokenizer.encode(
train_text
)
val_bpe_ids = tokenizer.encode(
val_text
)
test_bpe_ids = tokenizer.encode(
test_text
)
Convert the ID lists to long tensors, which is the type embedding layers expect for indices:
train_tensor = torch.tensor(
train_bpe_ids,
dtype=torch.long,
)
val_tensor = torch.tensor(
val_bpe_ids,
dtype=torch.long,
)
test_tensor = torch.tensor(
test_bpe_ids,
dtype=torch.long,
)
Save them to their own directory:
data/
└── encoded_bpe/
├── train_ids.pt
├── val_ids.pt
└── test_ids.pt
Leave the character-encoded files untouched; keeping both makes later comparison easy and avoids feeding one model the other's IDs.
Validating the encoded splits
First, every ID must fall in the range from zero up to, but excluding, the vocabulary size:
for split_name, token_ids in [
("train", train_tensor),
("validation", val_tensor),
("test", test_tensor),
]:
assert token_ids.min().item() >= 0
assert (
token_ids.max().item()
< tokenizer.vocab_size
)
print(
split_name,
token_ids.shape,
token_ids.min().item(),
token_ids.max().item(),
)
Second, a sample from each split should round-trip back to the original text:
for text in [
train_text[:1_000],
val_text[:1_000],
test_text[:1_000],
]:
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded)
assert decoded == text
Plugging BPE into the model
The model architecture needs no structural changes. Only the vocabulary size in the configuration moves:
config = MiniGPTConfig(
vocab_size=bpe_tokenizer.vocab_size,
block_size=128,
embedding_dim=128,
num_heads=4,
num_layers=4,
expansion_factor=4,
dropout=0.1,
)
Point the data loading at the new files:
train_ids = torch.load(
"data/encoded_bpe/train_ids.pt"
)
From here on, the model predicts BPE tokens rather than characters.
Old checkpoints will not load
A character model typically has a vocabulary in this range:
100 to 300 character tokens
A BPE model will usually have something like:
500 to 2,000 subword tokens
Both the token embedding and the output head depend on vocabulary size, so their shapes change. A character checkpoint has:
token embedding: [character_vocab_size, C]
While the BPE model expects:
token embedding: [bpe_vocab_size, C]
Here C is the embedding dimension. The shapes do not match, so the old weights cannot be loaded directly. And even if they did, IDs mean different things under each tokenizer. You need to train a fresh model, and it helps to keep checkpoints apart:
checkpoints_char/
checkpoints_bpe/
Or, equivalently:
checkpoints/char/
checkpoints/bpe/
Fingerprinting the tokenizer
A shape mismatch is loud; pairing a checkpoint with a different BPE tokenizer of the same size is silent, because every ID just maps to the wrong text. The fingerprint hashes a compact JSON form of the base tokens and merges with SHA-256, so any change produces a new value.
import hashlib
def tokenizer_fingerprint(tokenizer):
payload = json.dumps(
{
"base_tokens": (
tokenizer.base_tokens
),
"merges": [
list(pair)
for pair in tokenizer.merges
],
},
ensure_ascii=False,
separators=(",", ":"),
).encode("utf-8")
return hashlib.sha256(
payload
).hexdigest()
Store the fingerprint inside each checkpoint:
checkpoint[
"tokenizer_fingerprint"
] = tokenizer_fingerprint(
tokenizer
)
At load time, compute the fingerprint of the active tokenizer and refuse to continue if it does not match the one saved with the checkpoint.
The performance limitation you should know about
The encoder replays every merge rule across the entire token sequence:
for pair in self.merges:
tokens = merge_pair(...)
That is one full pass over the text per merge, so large texts with thousands of merges encode slowly. The implementation deliberately favors clarity, correctness and inspectability over speed. Production tokenizers pre-split text into words, cache repeated segmentations and apply merges by rank; for learning, the transparent version wins.
The complete program
Put everything together in a single script:
build_bpe_tokenizer.py
The full program below combines the pair helpers, the tokenizer class, validation helpers and a command-line entry point. Two details differ from the snippets: saving now stores the fingerprint and writes through a temporary file that is renamed over the target, so an interrupted run never leaves a half-written file, and training verifies that the base vocabulary covers the training text. The main function trains on the configured prefix, saves and reloads the tokenizer, checks fingerprints, round-trips and saves every split, validates ID ranges, and prints statistics.
One line in the "longest learned tokens" loop appears to have been mangled: f"{loaded_tokenizer.stoi5d} " will raise an AttributeError, because stoi5d does not exist. The intended expression is almost certainly f"{loaded_tokenizer.stoi[token]:5d} ", which prints each token's ID padded to five characters. Fix that line before running the script.
import argparse
import hashlib
import json
from collections import Counter
from pathlib import Path
import torch
# ==================================================
# Pair Operations
# ==================================================
def count_pairs(tokens):
return Counter(
zip(tokens[:-1], tokens[1:])
)
def merge_pair(
tokens,
pair,
merged_token,
):
output = []
index = 0
while index < len(tokens):
if (
index < len(tokens) - 1
and tokens[index] == pair[0]
and tokens[index + 1] == pair[1]
):
output.append(merged_token)
index += 2
else:
output.append(tokens[index])
index += 1
return output
# ==================================================
# BPE Tokenizer
# ==================================================
class BPETokenizer:
def __init__(
self,
base_tokens,
merges,
):
self.base_tokens = list(
base_tokens
)
self.merges = [
tuple(pair)
for pair in merges
]
self.tokens = list(
self.base_tokens
)
for left, right in self.merges:
merged_token = left + right
if merged_token not in self.tokens:
self.tokens.append(
merged_token
)
self.stoi = {
token: index
for index, token
in enumerate(self.tokens)
}
self.itos = {
index: token
for token, index
in self.stoi.items()
}
self.vocab_size = len(
self.tokens
)
@classmethod
def train(
cls,
text,
target_vocab_size,
minimum_pair_frequency=2,
base_tokens=None,
verbose=True,
):
if not text:
raise ValueError(
"Training text cannot be empty"
)
if base_tokens is None:
base_tokens = sorted(
set(text)
)
else:
base_tokens = sorted(
set(base_tokens)
)
missing_characters = (
set(text) - set(base_tokens)
)
if missing_characters:
raise ValueError(
"Base vocabulary is missing "
"training characters"
)
if (
target_vocab_size
< len(base_tokens)
):
raise ValueError(
"target_vocab_size cannot be "
"smaller than the number of "
"base tokens"
)
working_tokens = list(text)
vocabulary = set(base_tokens)
merges = []
print(
"Initial characters:",
len(working_tokens),
)
print(
"Base vocabulary size:",
len(base_tokens),
)
while (
len(vocabulary)
< target_vocab_size
):
pair_counts = count_pairs(
working_tokens
)
if not pair_counts:
print(
"No adjacent pairs remain."
)
break
best_pair, frequency = (
pair_counts.most_common(1)[0]
)
if (
frequency
< minimum_pair_frequency
):
print(
"Stopping because the most "
"frequent pair occurs only "
f"{frequency} time(s)."
)
break
merged_token = "".join(
best_pair
)
working_tokens = merge_pair(
tokens=working_tokens,
pair=best_pair,
merged_token=merged_token,
)
merges.append(best_pair)
vocabulary.add(merged_token)
merge_number = len(merges)
if (
verbose
and (
merge_number <= 20
or merge_number % 100 == 0
)
):
print(
f"Merge {merge_number:4d}: "
f"{best_pair!r} "
f"→ {merged_token!r}, "
f"frequency={frequency:,}, "
f"sequence length="
f"{len(working_tokens):,}, "
f"vocabulary="
f"{len(vocabulary):,}"
)
tokenizer = cls(
base_tokens=base_tokens,
merges=merges,
)
print(
"Final vocabulary size:",
tokenizer.vocab_size,
)
print(
"Final training sequence length:",
len(working_tokens),
)
return tokenizer
@classmethod
def from_file(cls, path):
with open(
path,
"r",
encoding="utf-8",
) as file:
data = json.load(file)
return cls(
base_tokens=data[
"base_tokens"
],
merges=data["merges"],
)
def encode_tokens(self, text):
unknown_characters = sorted({
char
for char in text
if char not in self.stoi
})
if unknown_characters:
raise ValueError(
"Text contains characters "
"missing from the base vocabulary: "
f"{unknown_characters}"
)
tokens = list(text)
for left, right in self.merges:
tokens = merge_pair(
tokens=tokens,
pair=(left, right),
merged_token=left + right,
)
return tokens
def encode(self, text):
return [
self.stoi[token]
for token in self.encode_tokens(
text
)
]
def decode(self, token_ids):
return "".join(
self.itos[int(token_id)]
for token_id in token_ids
)
def save(self, path):
path = Path(path)
path.parent.mkdir(
parents=True,
exist_ok=True,
)
data = {
"type": "character_bpe",
"base_tokens": (
self.base_tokens
),
"merges": [
[left, right]
for left, right
in self.merges
],
"vocab_size": (
self.vocab_size
),
"fingerprint": (
self.fingerprint()
),
}
temporary_path = path.with_suffix(
".tmp"
)
with open(
temporary_path,
"w",
encoding="utf-8",
) as file:
json.dump(
data,
file,
ensure_ascii=False,
indent=2,
)
temporary_path.replace(path)
def fingerprint(self):
payload = json.dumps(
{
"base_tokens": (
self.base_tokens
),
"merges": [
list(pair)
for pair in self.merges
],
},
ensure_ascii=False,
separators=(",", ":"),
).encode("utf-8")
return hashlib.sha256(
payload
).hexdigest()
# ==================================================
# Validation
# ==================================================
def validate_round_trip(
tokenizer,
text,
split_name,
):
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded)
if decoded != text:
raise AssertionError(
f"Round-trip failed for "
f"{split_name}"
)
print(
f"{split_name} round-trip passed: "
f"{len(text):,} characters "
f"→ {len(encoded):,} tokens"
)
return encoded
def save_token_tensor(
token_ids,
path,
):
path = Path(path)
path.parent.mkdir(
parents=True,
exist_ok=True,
)
tensor = torch.tensor(
token_ids,
dtype=torch.long,
)
temporary_path = path.with_suffix(
".tmp"
)
torch.save(
tensor,
temporary_path,
)
temporary_path.replace(path)
return tensor
# ==================================================
# Arguments
# ==================================================
def parse_args():
parser = argparse.ArgumentParser(
description=(
"Train a simple educational BPE "
"tokenizer on WikiText-2"
)
)
parser.add_argument(
"--processed-dir",
type=str,
default="data/processed",
)
parser.add_argument(
"--tokenizer-output",
type=str,
default=(
"tokenizer/bpe_tokenizer.json"
),
)
parser.add_argument(
"--encoded-output-dir",
type=str,
default="data/encoded_bpe",
)
parser.add_argument(
"--target-vocab-size",
type=int,
default=500,
)
parser.add_argument(
"--training-characters",
type=int,
default=500_000,
)
parser.add_argument(
"--minimum-pair-frequency",
type=int,
default=2,
)
return parser.parse_args()
# ==================================================
# Main Program
# ==================================================
def main():
args = parse_args()
processed_dir = Path(
args.processed_dir
)
train_path = (
processed_dir
/ "wikitext2_train_clean.txt"
)
val_path = (
processed_dir
/ "wikitext2_val_clean.txt"
)
test_path = (
processed_dir
/ "wikitext2_test_clean.txt"
)
for path in [
train_path,
val_path,
test_path,
]:
if not path.exists():
raise FileNotFoundError(
f"Required file not found: "
f"{path}"
)
train_text = train_path.read_text(
encoding="utf-8"
)
val_text = val_path.read_text(
encoding="utf-8"
)
test_text = test_path.read_text(
encoding="utf-8"
)
print(
"Training characters:",
len(train_text),
)
print(
"Validation characters:",
len(val_text),
)
print(
"Test characters:",
len(test_text),
)
all_characters = sorted(
set(
train_text
+ val_text
+ test_text
)
)
training_character_count = min(
args.training_characters,
len(train_text),
)
bpe_training_text = train_text[
:training_character_count
]
print(
"\nBPE training characters:",
len(bpe_training_text),
)
tokenizer = BPETokenizer.train(
text=bpe_training_text,
target_vocab_size=(
args.target_vocab_size
),
minimum_pair_frequency=(
args.minimum_pair_frequency
),
base_tokens=all_characters,
verbose=True,
)
tokenizer.save(
args.tokenizer_output
)
print(
"\nTokenizer saved to:",
args.tokenizer_output,
)
print(
"Tokenizer fingerprint:",
tokenizer.fingerprint(),
)
loaded_tokenizer = (
BPETokenizer.from_file(
args.tokenizer_output
)
)
assert (
loaded_tokenizer.fingerprint()
== tokenizer.fingerprint()
)
print(
"Tokenizer reload test passed."
)
print(
"\nEncoding dataset splits..."
)
train_ids = validate_round_trip(
tokenizer=loaded_tokenizer,
text=train_text,
split_name="Training",
)
val_ids = validate_round_trip(
tokenizer=loaded_tokenizer,
text=val_text,
split_name="Validation",
)
test_ids = validate_round_trip(
tokenizer=loaded_tokenizer,
text=test_text,
split_name="Test",
)
output_dir = Path(
args.encoded_output_dir
)
train_tensor = save_token_tensor(
train_ids,
output_dir / "train_ids.pt",
)
val_tensor = save_token_tensor(
val_ids,
output_dir / "val_ids.pt",
)
test_tensor = save_token_tensor(
test_ids,
output_dir / "test_ids.pt",
)
for split_name, tensor in [
("Training", train_tensor),
("Validation", val_tensor),
("Test", test_tensor),
]:
assert tensor.min().item() >= 0
assert (
tensor.max().item()
< loaded_tokenizer.vocab_size
)
print(
f"{split_name} tensor: "
f"shape={tuple(tensor.shape)}, "
f"min={tensor.min().item()}, "
f"max={tensor.max().item()}"
)
character_tokens = len(
train_text
)
bpe_tokens = len(
train_ids
)
compression_factor = (
character_tokens / bpe_tokens
)
reduction_percentage = (
1.0
- bpe_tokens / character_tokens
) * 100.0
print(
"\nCharacter token count:",
f"{character_tokens:,}",
)
print(
"BPE token count:",
f"{bpe_tokens:,}",
)
print(
"Compression factor:",
f"{compression_factor:.2f}x",
)
print(
"Token-count reduction:",
f"{reduction_percentage:.2f}%",
)
learned_tokens = (
loaded_tokenizer.tokens[
len(
loaded_tokenizer.base_tokens
):
]
)
longest_tokens = sorted(
learned_tokens,
key=len,
reverse=True,
)
print("\nLongest learned tokens:")
for token in longest_tokens[:30]:
print(
f"{loaded_tokenizer.stoi5d} "
f"{token!r}"
)
sample = (
"Artificial intelligence "
"and machine learning"
)
unknown = [
char
for char in sample
if char not in (
loaded_tokenizer.stoi
)
]
if not unknown:
sample_tokens = (
loaded_tokenizer.encode_tokens(
sample
)
)
sample_ids = (
loaded_tokenizer.encode(
sample
)
)
print("\nSample text:")
print(sample)
print("\nSample BPE tokens:")
print(sample_tokens)
print("\nSample token IDs:")
print(sample_ids)
print("\nDecoded sample:")
print(
loaded_tokenizer.decode(
sample_ids
)
)
print(
"\nBPE tokenizer pipeline "
"completed successfully."
)
if __name__ == "__main__":
main()
Running it
The default configuration is a small, educational run:
python build_bpe_tokenizer.py
To learn a larger vocabulary:
python build_bpe_tokenizer.py \
--target-vocab-size 1000
To train on more text as well:
python build_bpe_tokenizer.py \
--training-characters 1000000 \
--target-vocab-size 1000
For a fast smoke test that just confirms the pipeline works:
python build_bpe_tokenizer.py \
--training-characters 100000 \
--target-vocab-size 300
If your shell does not support backslash line continuations, write each command on a single line.
What the output looks like
A run might print something like this:
Training characters: 10,000,000
Validation characters: 1,100,000
Test characters: 1,200,000
BPE training characters: 500,000
Initial characters: 500,000
Base vocabulary size: 200
Merge 1: (' ', 't') → ' t', frequency=19,842
Merge 2: ('h', 'e') → 'he', frequency=17,415
Merge 3: (' t', 'he') → ' the', frequency=14,108
...
Final vocabulary size: 500
The first merges are exactly what you would expect for English: a space followed by t, then he, then those two combined into " the". After encoding, the summary looks similar to:
Character token count: 10,000,000
BPE token count: 4,500,000
Compression factor: 2.22x
Token-count reduction: 55.00%
Your numbers will vary. Note that round-tripping the full training split with this encoder can dominate runtime at larger vocabularies.
What changes for the model and its context window
Token counts on a real sentence
Take this sentence:
The history of artificial intelligence
A character tokenizer produces roughly this many tokens:
38 tokens
A BPE tokenizer might produce whole-word tokens like these:
["The", " history", " of", " artificial", " intelligence"]
Or it might produce smaller fragments, depending on which merges it learned. Either way, the count lands in a range like:
5 to 15 tokens
The segmentation is not designed by hand; it falls out of the statistics of the training corpus.
More text per context window
Suppose the model uses this block size:
block_size = 128
With characters, 128 tokens cover about 128 characters. With BPE, the same 128 tokens can cover several hundred characters. The model sees more linguistic context without making the attention computation any larger. That matters because the cost of self-attention grows roughly with:
T²
Here T is the sequence length, so doubling T roughly quadruples attention cost, while better tokenization packs more text into the same T.
The vocabulary size trade-off
A bigger BPE vocabulary generally means shorter sequences, but it also makes certain matrices bigger. The token embedding table holds this many parameters:
vocab_size × embedding_dim
With weight tying, the output head reuses the same matrix rather than adding a second one. Compare a small vocabulary:
Vocabulary: 500
Embedding dimension: 128
Embedding parameters: 64,000
With a larger one:
Vocabulary: 8,000
Embedding dimension: 128
Embedding parameters: 1,024,000
So a bigger vocabulary shortens sequences but raises embedding parameters, the cost of output logits, memory use and tokenizer training time. Each extra token also needs enough occurrences to learn a good embedding, so on a corpus the size of WikiText-2 a huge vocabulary yields many barely-trained tokens. The right size depends on dataset and model.
Retraining the model on BPE tokens
Update the training script to load the tokenizer file:
tokenizer/bpe_tokenizer.json
And the encoded splits:
data/encoded_bpe/train_ids.pt
data/encoded_bpe/val_ids.pt
data/encoded_bpe/test_ids.pt
Then build the configuration from the tokenizer's vocabulary size:
config = MiniGPTConfig(
vocab_size=bpe_tokenizer.vocab_size,
block_size=128,
embedding_dim=128,
num_heads=4,
num_layers=4,
expansion_factor=4,
dropout=0.1,
)
And write checkpoints to their own directory:
checkpoints/bpe/
Apart from these changes, the training loop can stay essentially as it was.
Comparing the two models fairly
For a meaningful comparison, keep architecture (width, depth, heads), data, optimizer, schedule, training budget and evaluation method fixed where you can. Even then, comparing at equal training steps is misleading, because a batch of BPE tokens contains considerably more text than a batch of characters. Better axes for comparison are:
Total characters processed
Total bytes processed
Training time
Compute budget
Perplexity is per prediction unit, so character and BPE perplexities cannot be compared directly. For one comparable number, convert both to bits per character; otherwise judge each model on generation quality, compute cost and tokenizer-appropriate metrics.
Common mistakes
Building a separate tokenizer for each split
This is wrong:
Train tokenizer from training text
Validation tokenizer from validation text
Test tokenizer from test text
Training, validation and test data must all be encoded by one and the same tokenizer; otherwise the same ID means different things in different splits.
Applying merges out of order
Merges build on each other, so they must be replayed in exactly the order they were learned:
for pair in self.merges:
apply(pair)
Sorting the rules, for example alphabetically for readability, changes how text is segmented.
Overwriting the character-encoded files
Keep the BPE IDs in their own directory:
data/encoded_bpe/
And leave the character IDs where they were:
data/encoded/
Learning merges from validation text
Merge statistics should come from training text only. Pulling the base character set from every split is fine for coverage, but merge frequencies must reflect the training corpus alone, or your validation numbers will be optimistic.
Expecting learned tokens to be whole words
BPE happily learns fragments such as:
"ing"
"tion"
" th"
" arti"
Subword tokens follow frequency, not human word boundaries, and that is expected behavior.
Assuming a bigger vocabulary is always better
Bigger vocabularies cost parameters and create rare, poorly trained tokens. Try several sizes and measure. Likewise, never compare raw perplexity across tokenizers.
Exercises
- Compare three vocabulary sizes. Train with the sizes below and compare training time, sequence length, compression, longest tokens and embedding size.
300 tokens
500 tokens
1,000 tokens
- Study the early merges. Print the first 50 merge rules. Do whitespace patterns appear? Common suffixes? Punctuation combinations? Which merges surprise you?
- Tokenize one sentence both ways. Encode the following sentence with both the character tokenizer and the BPE tokenizer, and compare the tokens, IDs, token count and decoded result.
Artificial intelligence is transforming technology.
- Probe rare words. Encode the following words and observe how BPE assembles unfamiliar words from known pieces.
electromagnetism
neurocomputational
uncharacteristically
- Measure context coverage. Decode the first 128 tokens under each tokenizer and see how much text each window spans.
- Train a BPE model. Train on the directory below and compare its samples against the character model at equal compute.
data/encoded_bpe/
- Enforce fingerprints. Store the tokenizer fingerprint in every checkpoint and refuse to run inference when the checkpoint and the active tokenizer disagree.
- Speed up encoding. Replace the one-pass-per-merge encoder with something faster, drawing on merge ranks, heaps, linked token lists, per-word caches or tries.
Tokenization rules worth keeping
- Subwords sit between flexible-but-long characters and compact-but-brittle words.
- A BPE tokenizer is fully defined by its base tokens plus its ordered merges; save and fingerprint exactly that.
- Encoding replays merges in order, decoding concatenates, and round-trips are lossless.
- One tokenizer serves every split, and merges are learned from training data only.
- A new tokenizer means new token files, a new vocabulary size and a freshly trained model.
- Vocabulary size trades shorter sequences for bigger embedding and output layers, and perplexity does not transfer across tokenizers.
The full pipeline, from raw text to a retrained model, looks like this:
WikiText training text
→ character vocabulary
→ pair frequencies
→ repeated merge operations
→ BPE vocabulary and merge rules
→ encode all splits
→ save token tensors
→ train a new Mini-GPT
The broader point is that tokenization is not a preprocessing detail. It decides which units the model learns, how much text fits in its context window and how large its output prediction problem is. Once a small GPT works on BPE tokens, the natural next step is to see how the same next-token machinery grows into modern generative AI systems:
Next-token predictor
→ pretrained language model
→ instruction-following model
→ retrieval-augmented application
→ tool-using GenAI system