Chunk Size and Overlap in RAG: Stride, Token Limits and Seams
How to chunk a long book for RAG: choose size and overlap, understand stride, stay under the embedding token limit, store metadata and see overlap in a real split.
This article is published in English.
Take Benjamin Franklin's autobiography from Project Gutenberg. Pasting the whole book into a prompt is wasteful even if the context window could hold it. A question like why Franklin wanted Britain to keep Canada probably has its answer in two or three passages, and the remaining pages would eat context and bury the evidence. Retrieval-augmented generation (RAG) exists for this situation. The system locates the passages that bear on the question and passes only those to the language model. The book stays in storage, and retrieval picks the evidence.
Embeddings and why text gets cut up
RAG starts by converting pieces of text into embeddings, numeric representations of meaning. Passages that mean similar things land close together in a high-dimensional vector space, even with different wording. When a question arrives, it's embedded as well, the nearest stored vectors are found, and the matching source passages come back. The vector locates the text, and the text is what the model reads.
One embedding can't represent a whole book. A single vector for Franklin's life would blur a Boston boyhood, printing, electricity experiments, Poor Richard and testimony before Parliament into something that describes none of them well. So the text has to be chunked before embedding.
Size, overlap and stride
A reasonable starting point:
DEFAULT_CHUNK_SIZE = 800 # characters
DEFAULT_CHUNK_OVERLAP = 150 # characters
People often treat these as two unrelated knobs. They really define one thing, how far the window moves forward each step:
stride = chunk_size − chunk_overlap
= 800 − 150
= 650 characters
That stride is the amount of fresh text each new chunk adds.
Treat chunk size as a ceiling and not a goal. Splitters prefer to cut at a natural seam, whether paragraph, line, sentence or word, so asking for 800 characters can produce chunks of 640, 731 or a full 800.
Overlap is text shared by neighboring chunks, and it does real work. With none, a cut can leave a stray fragment like this:
"…but left school at the age of ten."
On its own, that line says little. With overlap, the following chunk keeps enough earlier context to stay readable and findable. It matters most for long sentences and tightly linked arguments. Franklin's letter on Canada has a long chain of clauses that could straddle a boundary, leaving one chunk without its conclusion and the next without its premise. A modest overlap keeps the link in both records.
The price is duplication. Neighbors that share wording may be retrieved together, which is normal. When duplicates push out better evidence, a diversity-aware method such as maximal marginal relevance (MMR) helps.
What each stored chunk holds
Every chunk keeps four fields:
id
document
embedding
metadata
An example record:
id 7c2a1e90-4b11-4d3e-9f08-12a6c0e84b21
document "His boyhood in Boston was a stern beginning of the habit
of hard work and rigid economy which marked the man. For
a year he went to the Latin Grammar School on School
Street, but left off at the age of ten to help his father
in making soap and candles."
embedding 384 floats — [-0.0412, 0.0187, 0.0621, -0.0094, 0.0330, …]
norm = 1.0
metadata {
book: "Franklin's Autobiography",
source: "https://www.gutenberg.org/cache/epub/36151/pg36151-images.html",
gutenberg_id: 36151,
page_label: "5",
chunk_index: 2
}
The embedding serves search, and the original text serves reading, quoting and citing. Metadata deserves care too. If the edition prints page labels such as [Pg 5], keep them, so a citation points to a page readers can find in their copy.
Strip out low-value material before indexing: title pages, boilerplate, license text, repeated headers and footnote dumps. These produce weak embeddings and water down retrieval.
Normalized vectors
Embedding vectors are usually L2-normalized, which scales each to length 1:
‖v‖ = √(v₁² + v₂² + … + vₙ²)
Cosine similarity compares two vectors as follows:
cos θ = (a · b) / (‖a‖ ‖b‖)
When both have unit length, that collapses into a plain dot product:
If ‖a‖ = ‖b‖ = 1:
cos θ = a · b
Ranking therefore reflects how closely two meanings point the same way, and raw vector magnitude plays no part, which keeps retrieval quick and meaningful.
Tokens, not characters
Settings in characters can hide a hard limit, because embedding models count tokens. The all-MiniLM-L6-v2 model accepts at most 256 tokens. A longer chunk may be truncated without any warning. The database still holds the full text, but its embedding covers only the opening part, and the cut-off tail becomes hard or impossible to retrieve. A cautious estimate:
maximum safe chunk size in characters
≈ model token limit × 4
For a 256-token model that gives roughly 1,000 characters. Using four characters per token leaves headroom, since dialogue, abbreviations, punctuation and formatted text can tokenize more densely than prose.
Newer models accept much longer inputs, but chunking still matters. A bigger window doesn't solve retrieval, because overly broad chunks mix unrelated topics into one fuzzy vector.
Walking the pipeline
First, chunk the loaded pages:
from rag_qa.gutenberg import load_pages
from rag_qa.chunk import chunk_documents
from rag_qa.config import load_settings
s = load_settings()
print(s.summary())
# {
# 'chunk_size_chars': 800,
# 'chunk_overlap_chars': 150,
# 'stride_chars': 650, # size − overlap; this sets chunk count
# 'model': 'all-MiniLM-L6-v2',
# 'model_max_tokens': 256, # hard cap; overflow is silent
# 'embedding_dims': 384,
# 'max_safe_chunk_chars': 1024, # 256 × 4
# 'book': "Franklin's Autobiography",
# 'gutenberg_id': 36151,
# }
pages = load_pages()
chunks = chunk_documents(pages, s.chunk_size, s.chunk_overlap)
print(len(pages), len(chunks), s.stride)
Second, check each chunk against the model's token limit:
from sentence_transformers import SentenceTransformer
from rag_qa.tokens import check_chunk
m = SentenceTransformer("all-MiniLM-L6-v2")
print(m.max_seq_length) # 256
for c in chunks:
r = check_chunk(c)
if r.truncated:
print("silent truncate:", r.n_chars, "chars /", r.n_tokens, "tokens")
The script prints the model's maximum length, then lists only the chunks that would be cut silently. A usual run prints just the limit:
256
When everything fits, that's all you see. If some chunks exceed it, you get extra lines:
256
silent truncate: 1820 chars / 412 tokens
silent truncate: 960 chars / 301 tokens
How to read those lines:
Third, look at what ends up stored:
from rag_qa.store import ingest, open_store
store = open_store()
ingest(chunks, store)
row = store.get(limit=1)
print(row["ids"][0])
print(row["documents"][0][:200])
print(len(row["embeddings"][0]), row["metadatas"][0])
# 384 floats, norm 1.0, metadata.page_label == printed [Pg N]
That snippet peeks at a single chunk after ingestion and prints three lines:
doc-12-p3-c0
She left school at the age of ten. The next sentence continues on the same page…
384 {'page_label': '3', 'source': 'notes.pdf'}
Fourth, query it:
from rag_qa.retrieve import search, search_mmr
q = "why did Franklin want Britain to keep Canada"
for hit in search(q, k=5):
print(f"{hit['score']:.3f} p.{hit['metadata']['page_label']} {hit['document'][:120]}")
# overlap makes near-duplicate hits; MMR trades a little score for diversity
for hit in search_mmr(q, k=5):
print(hit["metadata"]["page_label"], hit["score"])
A sample of the results:
0.812 p.7 I have long been of opinion that the foundations of the future grandeur and stability of the British empire lie in America
0.781 p.7 they are, nevertheless, broad and strong enough to support the greatest political structure that human wisdom ever yet
0.744 p.7 I am, therefore, by no means for restoring Canada. If we keep it all the country from the St. Lawrence to the Mississippi
0.691 p.8 I left England about the end of August, 1762, in company with ten sail of merchant ships
0.640 p.6 In this Autobiography Franklin tells of his own life to the year 1757, when he went to England
The top few hits often come from the same page. That's overlap at work, since adjacent chunks of one clause all match.
Overlap shown on a small sample
Take a short passage of about 190 characters from the book's introduction, describing the author's Boston boyhood: early habits of hard work and thrift, a year at the Latin Grammar School on School Street, and leaving at age ten. Split it with a chunk size of only 60.
With no overlap, the splitter cuts exactly at the limit:
[0] 59 His boyhood in Boston was a stern beginning of the habit of
[1] 55 hard work and rigid economy which marked the man. For a
[2] 58 year he went to the Latin Grammar School on School Street,
[3] 31 but left off at the age of ten.
Chunk 0 ends on "the habit of" and chunk 1 begins at "hard work", with nothing connecting them. The phrase "habit of hard work" never sits whole in any chunk. If the cut falls mid-sentence, each vector encodes half an idea, and a query needing the full sentence matches both halves badly.
With an overlap of 15 characters:
[0] 59 His boyhood in Boston was a stern beginning of the habit of
[1] 57 ⟦the habit of⟧ hard work and rigid economy which marked the
[2] 55 ⟦marked the⟧ man. For a year he went to the Latin Grammar
[3] 58 ⟦Latin Grammar⟧ School on School Street, but left off at the
[4] 22 ⟦off at the⟧ age of ten.
The marker below shows which text repeats from the previous chunk:
⟦…⟧ = text repeated from the previous chunk
No chunk reaches 60 characters, again showing that the size is only a ceiling. Skipping overlap gives a smaller index and no duplicate hits, but anything the splitter happened to slice through is lost. Overlap insures those seams and isn't extra context for its own sake.
The four jobs and the tools for them
RAG is four tasks: load text, cut it, embed it, and store and search the vectors. No single library is "RAG". Tools for cutting and orchestration:
Embedding models:
Popular vector databases complete the stack.