This article is published in English.
Practical notes: Chunking is the hidden design decision in RAG
Operable walkthrough of Practical notes: Chunking is the hidden design decision in RAG: contracts, checks, and drop-in code slots for teams shipping this pattern.
This walkthrough rebuilds the path from raw materials to a working system for: Chunking is the hidden design decision in RAG. The focus is operable steps, explicit checks, and code that you can drop into a repo without guessing intent. For the Overview stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.
The RAG flow
When working through the The RAG flow stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.
From text to vectors
When working through the From text to vectors stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.
Chunk size and overlap
When working through the Chunk size and overlap stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface. When working through the Chunk size and overlap stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.
DEFAULT_CHUNK_SIZE = 800 # characters
DEFAULT_CHUNK_OVERLAP = 150 # characters
stride = chunk_size − chunk_overlap
= 800 − 150
= 650 characters
"…but left school at the age of ten."
What a vector-store record contains
The What a vector-store record stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.
id
document
embedding
metadata
id 7c2a1e90-4b11-4d3e-9f08-12a6c0e84b21
document "His boyhood in Boston was a stern beginning of the habit
of hard work and rigid economy which marked the man. For
a year he went to the Latin Grammar School on School
Street, but left off at the age of ten to help his father
in making soap and candles."
embedding 384 floats — [-0.0412, 0.0187, 0.0621, -0.0094, 0.0330, …]
norm = 1.0
metadata {
book: "Franklin's Autobiography",
source: "https://www.gutenberg.org/cache/epub/36151/pg36151-images.html",
gutenberg_id: 36151,
page_label: "5",
chunk_index: 2
}
Why normalisation matters
The Why normalisation matters stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.
‖v‖ = √(v₁² + v₂² + … + vₙ²)
cos θ = (a · b) / (‖a‖ ‖b‖)
If ‖a‖ = ‖b‖ = 1:
cos θ = a · b
The token limit you cannot see
The The token limit you stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices. The The token limit you stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.
maximum safe chunk size in characters
≈ model token limit × 4
Code snippets
For the Code snippets stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.
from rag_qa.gutenberg import load_pages
from rag_qa.chunk import chunk_documents
from rag_qa.config import load_settings
s = load_settings()
print(s.summary())
# {
# 'chunk_size_chars': 800,
# 'chunk_overlap_chars': 150,
# 'stride_chars': 650, # size − overlap; this sets chunk count
# 'model': 'all-MiniLM-L6-v2',
# 'model_max_tokens': 256, # hard cap; overflow is silent
# 'embedding_dims': 384,
# 'max_safe_chunk_chars': 1024, # 256 × 4
# 'book': "Franklin's Autobiography",
# 'gutenberg_id': 36151,
# }
pages = load_pages()
chunks = chunk_documents(pages, s.chunk_size, s.chunk_overlap)
print(len(pages), len(chunks), s.stride)
from sentence_transformers import SentenceTransformer
from rag_qa.tokens import check_chunk
m = SentenceTransformer("all-MiniLM-L6-v2")
print(m.max_seq_length) # 256
for c in chunks:
r = check_chunk(c)
if r.truncated:
print("silent truncate:", r.n_chars, "chars /", r.n_tokens, "tokens")
256
256
silent truncate: 1820 chars / 412 tokens
silent truncate: 960 chars / 301 tokens
from rag_qa.store import ingest, open_store
store = open_store()
ingest(chunks, store)
row = store.get(limit=1)
print(row["ids"][0])
print(row["documents"][0][:200])
print(len(row["embeddings"][0]), row["metadatas"][0])
# 384 floats, norm 1.0, metadata.page_label == printed [Pg N]
doc-12-p3-c0
She left school at the age of ten. The next sentence continues on the same page…
384 {'page_label': '3', 'source': 'notes.pdf'}
from rag_qa.retrieve import search, search_mmr
q = "why did Franklin want Britain to keep Canada"
for hit in search(q, k=5):
print(f"{hit['score']:.3f} p.{hit['metadata']['page_label']} {hit['document'][:120]}")
# overlap makes near-duplicate hits; MMR trades a little score for diversity
for hit in search_mmr(q, k=5):
print(hit["metadata"]["page_label"], hit["score"])
0.812 p.7 I have long been of opinion that the foundations of the future grandeur and stability of the British empire lie in America
0.781 p.7 they are, nevertheless, broad and strong enough to support the greatest political structure that human wisdom ever yet
0.744 p.7 I am, therefore, by no means for restoring Canada. If we keep it all the country from the St. Lawrence to the Mississippi
0.691 p.8 I left England about the end of August, 1762, in company with ten sail of merchant ships
0.640 p.6 In this Autobiography Franklin tells of his own life to the year 1757, when he went to England
Addendum
For the Addendum stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap.
A. Same passage, overlap off and on
For the A Same passage overlap stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments. Cite the passages that actually grounded the answer. Without citations, operators cannot tell hallucination from an indexing gap. For the A Same passage overlap stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.
[0] 59 His boyhood in Boston was a stern beginning of the habit of
[1] 55 hard work and rigid economy which marked the man. For a
[2] 58 year he went to the Latin Grammar School on School Street,
[3] 31 but left off at the age of ten.
[0] 59 His boyhood in Boston was a stern beginning of the habit of
[1] 57 ⟦the habit of⟧ hard work and rigid economy which marked the
[2] 55 ⟦marked the⟧ man. For a year he went to the Latin Grammar
[3] 58 ⟦Latin Grammar⟧ School on School Street, but left off at the
[4] 22 ⟦off at the⟧ age of ten.
⟦…⟧ = text repeated from the previous chunk
B. Libraries for building RAG, and where vectors go
When working through the B Libraries for building stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Measure recall on a fixed question set before tuning prompts. Prompt churn rarely fixes a weak retrieval surface.
Operational checklist
The Operational checklist stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope.
Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph.
Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.
Add a smoke test that exercises the critical path in CI with fixtures, not live paid APIs, whenever budgets allow.
Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish.
Separate chunking policy from retrieval policy. Changing one should not force a rewrite of the other when quality metrics move.
Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.
Batch note for 62ec22cbf28d: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.