Token Budgets vs. Semantic Boundaries: Getting RAG Chunk Splits Right

A RAG pipeline can look healthy at every visible layer, with reasonable embeddings, plausible similarity scores and a well-populated vector store, and still hand the model fragments that stop mid-sentence. In many of those cases the defect sits in the ingestion code, where a chunking routine was built on a wrong assumption about how text turns into tokens. This guide separates the two ideas cleanly, shows where each one lives in the pipeline, and walks through a splitting approach that respects meaning first while keeping exact control over token budgets.

Engineers routinely say things like "chunk it into 512 tokens" or "token-based chunking", and the phrasing merges two different operations into one. They solve different problems, run at different stages and break in different ways. Knowing exactly where one stops and the other starts is what lets you reason about a retrieval system instead of nudging parameters until an evaluation score improves.

Two operations, two owners

The shortest useful definitions are these:

• Tokenization turns text into the discrete integer units a model actually consumes. It belongs to the model.
• Chunking divides documents into self-contained pieces that can be embedded, indexed and retrieved. It belongs to your pipeline and your data.

Tokenization runs every time any string reaches a model, whether you think about it or not. Chunking is a choice you make ahead of time about how to structure a corpus before a model ever sees it. The first is fixed and mostly invisible; the second is something you design, measure and refine over weeks.

The two interact all the time. A splitter that ignores how the tokenizer behaves will quietly produce poor chunks. Still, they are separate steps, and treating them as the same thing is the root of a surprising number of subtle retrieval bugs.

How a model reads text

No transformer LLM, whether GPT-4o, Claude, Llama or anything else in production, reads words. It reads a sequence of integers, each pointing into a fixed vocabulary learned during training. The tokenizer is the deterministic algorithm that converts raw text into that sequence. It is chosen once, when the model is trained, and ships as part of the model's artifacts. You cannot swap it per request.

Subword vocabularies: BPE, WordPiece, SentencePiece

Most current models rely on subword tokenization, typically from the BPE, WordPiece or SentencePiece families. The shared principle is to begin from characters or raw bytes and repeatedly merge the most frequent neighbouring pairs into new vocabulary entries until a target vocabulary size is reached, commonly somewhere from 32k to 128k entries or more.

What this means in day-to-day work:

• Frequent English words such as the, is or model usually map to one token each.
• Less frequent words break into subword pieces, so tokenization may become token plus ization.
• Unusual strings, including code identifiers, non-Latin scripts, emoji and invented product names, can fan out into many tokens per word and occasionally close to one token per character.

That is why a sentence in Hindi or Tamil can cost two to three times as many tokens as an English sentence with the same meaning, and why an identifier like xgboost_lstm_ensemble_v2 might consume around eight tokens when you planned for two. The exact ratios depend on the tokenizer, and newer vocabularies have narrowed the gap for some languages, so measure with the model you use. For any product serving non-English markets, for example an LLM layer that summarises Indian stock-exchange filings or brokerage order logs, this is a real cost and context-budget item rather than a curiosity.

Why token counts matter beyond the bill

Token counts do more than set the price of a request. They govern three things:

1. How much of the context window you use. A 128k window is measured in tokens, not words or characters. A chunk of 1,000 words has no fixed cost; its token count shifts with vocabulary, language and content type, since code, prose and numbers tokenize very differently.
2. What attention operates over. Transformers attend across token positions, so the way a passage is broken into tokens shapes how coherently the model can relate its parts. Pathological splits that hurt comprehension are uncommon, but certain encodings can produce them.
3. Where truncation happens. When input exceeds the window, it is cut at the token level, not at a sentence or chunk boundary, unless you manage that yourself. Careless truncation can leave a word half-finished.

Look at your own strings once

If you have never run your real production strings through OpenAI's tiktoken or Anthropic's tokenizer, it is worth doing a single time. Seeing your own prompt template broken into unexpected fragments turns an abstraction into something concrete. Markdown syntax, JSON braces and code fences are particular offenders: they tokenize less efficiently than ordinary prose.

The snippet below loads the cl100k_base encoding and counts the tokens in a typical shell command. Notice how flags and file extensions each tend to occupy two or three tokens, which is exactly the kind of content that inflates a budget without anyone noticing.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("pip-compile requirements.in --output-file requirements.txt")
print(len(tokens), tokens)
# CLI flags and file extensions often split into 2-3 tokens each —
# this is exactly the kind of string that silently inflates your token budget

How you decide what the model gets to see

Chunking lives one level up, in the pipeline rather than the model. It takes a large document, such as a PDF, a Confluence page, a code repository or a 40-page regulatory circular, and breaks it into smaller units that are embedded, indexed and retrieved independently. The chunk is the currency of the retrieval layer.

Two constraints make chunking necessary, and neither is really about tokenization:

1. Long embeddings get blurry. Pack too many unrelated ideas into one vector and it becomes an average of all of them, matching nothing precisely. A good chunk carries one coherent idea, not a whole document.
2. Context is still finite and costly. Even with windows of 128k tokens and beyond now common, placing the entire knowledge base in every prompt is not retrieval. It is a poor imitation of full-context inference, with more latency, more cost and weaker precision than a proper retrieval step.

Chunking strategies and how each one breaks

Fixed-size windows

The simplest approach cuts the text every N tokens or characters, usually with some overlap between neighbours. Most teams start here because it takes a few lines of code. The function below encodes the whole text, slices the token list into windows of chunk_size, decodes each window back to text and advances by chunk_size - overlap so that consecutive chunks share a margin.

def fixed_chunk(text, chunk_size=512, overlap=50):
    tokens = enc.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = start + chunk_size
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))
        start += chunk_size - overlap
    return chunks

The weakness is that this function has no concept of a sentence. It will place a boundary in the middle of a clause, separating a claim from the figure that supports it, or a code sample from the sentence that explains it. Those half-sentence chunks embed into vectors that sit between topics and match poorly, which is precisely the symptom described at the start. Two smaller caveats: overlap must stay below chunk_size or the loop never advances, and decoding an arbitrary token slice can cut a multibyte character in half, producing a replacement character at the edge of a chunk.

Recursive splitting on natural boundaries

A recursive splitter prefers paragraph breaks, falls back to sentence breaks, then word breaks, and only makes a hard cut when nothing else fits. LangChain's RecursiveCharacterTextSplitter and the node parsers in LlamaIndex follow this pattern. For prose-heavy corpora it is a far better default, because the document's own structure is honoured before the size limit is.

Semantic chunking

Rather than fixing a size, semantic chunking embeds individual sentences or small groups of them and places a boundary wherever the cosine similarity between neighbouring embeddings drops, which is where the topic changes. The resulting chunks hang together by meaning rather than by length. The price is an additional embedding pass across every document during ingestion.

Structure-aware chunking

Content with built-in structure, such as Markdown with headers, source files, API specifications or schema documentation, should be split along that structure. In code, a function definition is an obvious boundary. In a Markdown knowledge base, an ## section is one. For something like the README of a multi-agent system, splitting on section headers keeps far more retrievable meaning than any token count could. A deeper treatment of keeping supporting evidence together is in chunking that preserves evidence.

LLM-driven chunking

At the expensive end, you can ask an LLM to read the document and propose boundaries itself, aiming for pieces that each stand alone and answer a plausible question. It is slow and costly, but it earns its keep on high-value corpora such as regulatory filings or contracts, where a retrieval miss carries real consequences.

Where the two ideas collide

The interesting part is the point where these concepts meet, and it can be stated in one line: chunk size is normally expressed in tokens, but chunk boundaries should follow meaning, and those two goals pull in opposite directions.

Cutting strictly at the 512-token mark with chunk_size=512 satisfies the budget while ignoring meaning. Splitting only on paragraph breaks with no ceiling respects meaning but ignores that one paragraph may be 40 tokens and the next 4,000. The large one can exceed the embedding model's maximum sequence length, and the retrieved context ends up wildly uneven in how much information each chunk carries.

A sound splitter therefore handles both concerns, in a fixed order:

1. Find the semantic boundaries first: sections, paragraphs, sentences, code blocks.
2. Combine small neighbouring units until adding the next one would exceed the token budget.
3. Hard-split a single unit only when it is larger than the budget by itself, and when you do, tokenize it yourself so you decide where the cut lands instead of relying on a character-count guess that knows nothing about token boundaries.

The function below implements that order. It expects sections that have already been split on paragraphs or headers. For each one it measures the real token length; if adding it would overflow the current chunk, the accumulated sections are flushed as a chunk. A section that alone exceeds max_tokens is sliced on token boundaries; everything else is appended to the running chunk. Whatever remains at the end becomes the final chunk.

def chunk_by_tokens_with_semantic_bias(sections, max_tokens=512):
    chunks, current, current_len = [], [], 0
    for section in sections:  # pre-split by paragraph/header
        section_len = len(enc.encode(section))
        if current_len + section_len > max_tokens and current:
            chunks.append("\n".join(current))
            current, current_len = [], 0
        if section_len > max_tokens:
            # single section too big — hard split on token boundaries, not chars
            section_tokens = enc.encode(section)
            for i in range(0, len(section_tokens), max_tokens):
                chunks.append(enc.decode(section_tokens[i:i+max_tokens]))
        else:
            current.append(section)
            current_len += section_len
    if current:
        chunks.append("\n".join(current))
    return chunks

The key design decision is that size is measured with the target embedding model's tokenizer, not estimated from characters, and that a forced split can only ever fall on a token boundary the code controls. One refinement worth considering: the newline used in "\n".join adds a token or so per section that the running count does not include, so leave a little headroom below the model's hard limit.

Where each step sits in the pipeline

Put spatially, chunking is an upstream concern. It happens during ingestion, before any model is involved, and its output is stored. Tokenization is a downstream concern. It happens inside the model's input layer on every single request, including the embedding calls made during ingestion and the generation calls made at query time. Keeping that picture in mind tells you which code to open when something goes wrong.

Telling the two failure modes apart

When you triage a poor retrieval result, two symptoms look alike but point to different layers:

• The retrieved chunk is cut off or ends mid-thought. This is a chunking fault. The boundary logic split a semantic unit, and the fix belongs in the splitter, not in the choice of embedding model.
• The chunk looks complete, but the answer ignores half of it, or the prompt is unexpectedly truncated. This is a tokenization and context-budget fault. The token accounting is wrong somewhere, very often because tokens were estimated with len(text) / 4 and the real count came in higher.

Both regularly get blamed on "a bad embedding model". That is rarely the actual cause; boundary logic or token accounting almost always is.

Production checklist

When building or reviewing a RAG or agent pipeline, work through these points:

• Count tokens with the real tokenizer of the model you call. Use tiktoken for OpenAI models and the model-specific tokenizer for open-weight models. Never ship len(text) // 4 as the production counter.
• Measure chunk size in the embedding model's tokens. Embedding models often use a different tokenizer from the generation model and enforce a stricter maximum sequence length.
• Budget overlap explicitly. Overlap consumes part of the effective window, so include it in the arithmetic against the embedding model's input limit instead of learning about it from a truncation warning.
• Put structure before size. Default to structure-aware or recursive splitting, and reserve fixed-size windows for genuinely unstructured text.
• Log token counts and boundary decisions during ingestion. When quality drops, you want to see immediately whether a chunk was cut for size or for structure.
• Re-tokenize after switching embedding models. A strategy tuned against one vocabulary does not transfer cleanly; 512 tokens covers a different amount of text under a different tokenizer.

Wrapping up

Tokenization answers the model's question of how to read a string at all. Chunking answers your question of what the smallest self-contained unit worth retrieving is. The first is fixed the moment you pick a model; the second is a design surface entirely under your control, and it is usually where the real engineering leverage in a RAG system sits.

The two kinds of mistakes also fail differently. Token accounting errors show up as odd truncation and blown budgets, which are at least visible. Chunking errors produce a system that runs without a single error and still retrieves the wrong half of the right sentence, which is far harder to notice. Measure with the real tokenizer, split on meaning first, and log why every boundary was placed where it was.