Chunking is the whole retrieval system: Spring AI 800 vs 128 trap
When the splitter emits 800-token chunks but the embedder only reads 128, retrieval cannot see the truncated tail. Chunk size, window, and overlap are the retrieval system.
This article is published in English.
Spring AI’s default splitter cuts roughly 800-token chunks, while the bundled local embedding model only reads the first 128 tokens of each one. No
topK, similarity threshold, or reranker can surface text that never entered the vector.
The chunker decides what each vector is allowed to say. Every stage after the vector store only reorders candidates the chunker made searchable.
The pain point: eight hundred tokens in, one hundred twenty-eight embedded
Three lines of Spring AI 2.0.1 build the index behind many JVM RAG demos, and two hidden numbers decide what that index can find:
List<Document> pages = new TikaDocumentReader(handbook).read();
List<Document> chunks = new TokenTextSplitter().apply(pages);
vectorStore.add(chunks);
The first number lives in the splitter. new TokenTextSplitter() cuts near 800 tokens under OpenAI’s cl100k_base encoding, then backs up to the last ., ?, !, or newline if at least about 350 characters remain—roughly six hundred English words per chunk.
The second number lives in tokenizer JSON. When the store uses Spring AI’s local TransformersEmbeddingModel, it loads all-MiniLM-L6-v2 by default. The bundled tokenizer.json configures truncation like this:
"truncation": { "direction": "Right", "max_length": 128, "strategy": "LongestFirst", "stride": 0 }
DJL 0.36.0, the tokenizer layer under Spring AI 2.0.1, keeps truncation on and honors that max_length. Each chunk vector is therefore computed from the first 128 WordPiece tokens, two of which are [CLS] and [SEP]. Different tokenizers make the ratio approximate, not precise—and still lopsided. The vector roughly sees the first hundred words of a six-hundred-word chunk. The remainder is stored, retrieved, and passed to the chat model, yet it never participated in search.
Chunking is not mere preprocessing you set once before “real” retriever tuning. It decides what can be retrieved at all. Raising topK from a default of 4 returns more chunks whose tails are equally invisible and can dump thousands of tokens into the prompt. A cross-encoder reranker only reorders candidates vector search already returned; invisible text cannot be promoted. Similarity thresholds have little room left when SearchRequest already defaults to accepting everything at 0.0.
Switching to OpenAI text-embedding-3-small removes truncation because its input limit dwarfs the chunk, but quieter failure modes remain—covered below. Everything after the vector store can only reorder what the chunker allowed through.
Core architectural concepts
Four facts about how a chunk becomes a hit show up directly in Spring AI’s behavior.
A chunk is one vector, and one vector is an average
TransformersEmbeddingModel runs the ONNX model, reads last_hidden_state (384 dimensions per token), and mean-pools with the attention mask skipping padding. However long the chunk, search sees one vector:
v = Σ mᵢ·hᵢ / Σ mᵢ hᵢ = vector for token i, mᵢ = 1 for a real token, 0 for padding
score(question, chunk) = cos(q, v)
Mean pooling dilutes distinctive phrases buried after the truncation window even further: they never enter the sum. Retrieval scores compare the query vector against that average, so the leading tokens dominate both what is embedded and what similarity can notice.
Three tokenizers, and nothing compares them
Chunk boundaries, embedding inputs, and often the chat prompt each speak different token languages. A practical rule of thumb for local MiniLM setups is keeping embedding tokens inside the model window with room for specials:
chunk tokens (embedding tokenizer) ≤ window − 2 // 126 for the bundled MiniLM tokenizer
prompt tokens ≈ topK × chunk tokens + system + question + history
If the splitter counts cl100k_base while the embedder counts WordPiece, “800 tokens” is not “800 embedder tokens.” Teams that never compare the two ship indexes that look healthy in storage metrics and fail on questions answered only in chunk tails.
Your metadata is embedded or ignored, depending on one enum
Chunks carry metadata such as:
source: refund-policy.pdf
parent_document_id: 5b1f0c2e-8d7a-4c1e-9f3b-2a6d4e8c1b90
chunk_index: 3
total_chunks: 9
Annual plans are refunded pro rata from the day the request is received...
Whether those fields are concatenated into the embedded string or excluded depends on formatter configuration:
ContentFormatter formatter = DefaultContentFormatter.builder()
.withExcludedEmbedMetadataKeys("parent_document_id", "chunk_index", "total_chunks") // bookkeeping, not meaning
.build();
chunks.forEach(chunk -> chunk.setContentFormatter(formatter));
Embedding source paths or UUIDs can pollute semantic space; excluding headings can strip the only disambiguating signal. Choose embed versus retrieve metadata deliberately—one enum flip changes both recall and citation quality.
The default advisor drops your citations
A typical Q&A call looks simple:
ChatResponse response = chatClient.prompt()
.user(question)
.advisors(QuestionAnswerAdvisor.builder(vectorStore).build())
.call()
.chatResponse();
List<Document> used = response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);
List<String> citations = used.stream()
.map(d -> d.getMetadata().get("source") + " § " + d.getMetadata().getOrDefault("title", ""))
.distinct()
.toList();
Default advisors may pass context text to the model without preserving per-chunk source fields in the structured response. You retrieve the right paragraph and still cannot show the PDF page or heading unless you customize the advisor or post-process documents before prompting. Citations are a pipeline concern, not a UI flourish.
Hands-on implementation guide
Measure retrieval before you touch the splitter
Build a tiny evaluation set of questions paired with answer spans that must appear in retrieved text:
public record RetrievalCase(String question, String answerSpan) {}
public final class RetrievalEval {
private final VectorStore vectorStore;public RetrievalEval(VectorStore vectorStore) {
this.vectorStore = vectorStore;
}
public double recallAt(int k, List<RetrievalCase> cases) {
long hits = cases.stream().filter(c -> retrieved(c, k)).count();
return (double) hits / cases.size();
}
private boolean retrieved(RetrievalCase c, int k) {
List<Document> top = vectorStore.similaritySearch(
SearchRequest.builder().query(c.question()).topK(k).build());
String span = normalize(c.answerSpan());
return top.stream().anyMatch(d -> normalize(d.getText()).contains(span)); // a span split across chunks fails
}
private static String normalize(String s) {
return s == null ? "" : s.replaceAll("\\s+", " ").strip().toLowerCase(Locale.ROOT);
}
public static List<RetrievalCase> load(Path tsv) throws IOException {
return Files.readAllLines(tsv).stream()
.filter(line -> !line.isBlank())
.map(line -> line.split("\t", 2))
.map(p -> new RetrievalCase(p[0], p[1]))
.toList();
}
}
Run the current index, record hit-or-miss at a fixed topK, and only then change splitter settings. Without a baseline, every chunk-size tweak is storytelling.
Fit every chunk inside the embedding window
Guard documents so embedder length cannot silently truncate:
public final class EmbeddingWindowGuard implements DocumentTransformer {
private final HuggingFaceTokenizer tokenizer;
private final int window;
public EmbeddingWindowGuard(Path tokenizerJson, int window) throws IOException {
// truncation off, so we measure the real length rather than the clipped one
this.tokenizer = HuggingFaceTokenizer.newInstance(tokenizerJson, Map.of("truncation", "false"));
this.window = window;
}
public int count(String text) {
return tokenizer.encode(text).getIds().length; // includes [CLS] and [SEP]
}
@Override
public List<Document> apply(List<Document> chunks) {
for (Document chunk : chunks) {
int n = count(chunk.getText());
if (n > window) {
throw new IllegalStateException("%s chunk %s is %d tokens; the model reads %d".formatted(
chunk.getMetadata().get("source"), chunk.getMetadata().get("chunk_index"), n, window));
}
}
return chunks;
}
}
Configuration can raise related tokenizer options when appropriate:
spring.ai.embedding.transformer.tokenizer.options.maxLength=256
If the model only sees 128 tokens, either shrink chunks to fit or change models. Do not pretend a reranker will invent missing signal.
Split on structure first, size second
Prefer headings, paragraphs, and list boundaries before raw token counts:
public final class RecursiveTextSplitter extends TextSplitter {
private static final List<String> SEPARATORS = List.of("\n\n", "\n", ". ", " ");
private final ToIntFunction<String> tokens;
private final int maxTokens;
public RecursiveTextSplitter(ToIntFunction<String> tokens, int maxTokens) {
this.tokens = tokens; // the embedding model's tokenizer, not cl100k
this.maxTokens = maxTokens;
}
@Override
protected List<String> splitText(String text) {
return split(text, 0).stream().filter(s -> !s.isBlank()).toList();
}
private List<String> split(String text, int level) {
if (tokens.applyAsInt(text) <= maxTokens || level == SEPARATORS.size()) {
return List.of(text.strip());
}
String sep = SEPARATORS.get(level);
List<String> chunks = new ArrayList<>();
StringBuilder current = new StringBuilder();
for (String piece : text.split(Pattern.quote(sep))) {
String candidate = current.isEmpty() ? piece : current + sep + piece;
if (tokens.applyAsInt(candidate) <= maxTokens) {
current.setLength(0);
current.append(candidate);
continue;
}
if (!current.isEmpty()) {
chunks.add(current.toString().strip());
current.setLength(0);
}
if (tokens.applyAsInt(piece) <= maxTokens) {
current.append(piece);
} else {
chunks.addAll(split(piece, level + 1)); // only this piece drops to a finer boundary
}
}
if (!current.isEmpty()) {
chunks.add(current.toString().strip());
}
return chunks;
}
}
Then enforce the embedding window:
EmbeddingWindowGuard guard = new EmbeddingWindowGuard(Path.of("tokenizer.json"), 128);
TextSplitter splitter = new RecursiveTextSplitter(guard::count, 120); // headroom under the 126 usable
vectorStore.add(guard.apply(splitter.apply(sections)));
Structure-aware splits keep definitions with their terms and steps with their procedures—exactly the units users ask about.
Overlap buys one thing, and you pay for it everywhere
Overlap increases the chance a boundary phrase appears fully inside some chunk, at the cost of more chunks:
chunks ≈ ⌈ (N − o) / (s − o) ⌉
s = 800, o = 0 → about N / 800
s = 800, o = 400 → about N / 400 // twice the vectors, almost every sentence stored twice
Doubling overlap roughly doubles stored vectors and embed spend. Use modest overlap when boundaries cut sentences; do not treat large overlap as a substitute for fitting the embedding window.
Tables and code blocks break every strategy
Markdown tables and fenced code need specialized handling—row-wise splits or dedicated extractors—so a vector does not average an entire grid into mush:
static List<String> tableRows(String markdown) {
Parser parser = Parser.builder().extensions(List.of(TablesExtension.create())).build();
List<String> rows = new ArrayList<>();
parser.parse(markdown).accept(new AbstractVisitor() {
@Override
public void visit(CustomBlock block) {
if (!(block instanceof TableBlock table) || !(table.getLastChild() instanceof TableBody body)) {
visitChildren(block);
return;
}
List<String> headers = cells((TableRow) table.getFirstChild().getFirstChild());
for (Node row = body.getFirstChild(); row != null; row = row.getNext()) {
List<String> values = cells((TableRow) row);
StringJoiner line = new StringJoiner(" · ");
for (int i = 0; i < Math.min(headers.size(), values.size()); i++) {
line.add(headers.get(i) + ": " + values.get(i)); // every value keeps its column name
}
rws.add(line.toString());
}
}
});
return rows;
}
static List<String> cells(TableRow row) {
List<String> out = new ArrayList<>();
for (Node cell = row.getFirstChild(); cell != null; cell = cell.getNext()) {
StringBuilder text = new StringBuilder();
cell.accept(new AbstractVisitor() {
@Override public void visit(Text t) { text.append(t.getLiteral()); }
@Override public void visit(Code c) { text.append(c.getLiteral()); }
});
out.add(text.toString().strip());
}
return out;
}
Carry headings forward so a chunk that starts mid-section still knows its context:
public final class CarryHeadingForward implements DocumentTransformer {
@Override
public List<Document> apply(List<Document> documents) {
String heading = null;
for (Document doc : documents) {
Object title = doc.getMetadata().get("title");
if (title != null) {
heading = title.toString();
} else if (heading != null) {
doc.getMetadata().put("title", heading); // code blocks and the prose after them
}
}
return documents;
}
}
Without heading carry, retrieval returns orphan rows that the chat model cannot situate.
Embed small, return big
A useful pattern separates what you embed from what you return. Index compact sections for search, then expand to parent sections for the prompt:
public record Section(String id, String source, String heading, String body) {}
public List<Section> retrieve(String question) {
List<Document> children = vectorStore.similaritySearch(
SearchRequest.builder().query(question).topK(20).build()); // search wide on small chunks
Map<String, Double> bestByParent = new HashMap<>();
for (Document child : children) {
String parentId = (String) child.getMetadata().get("parent_document_id");
double score = child.getScore() == null ? 0.0 : child.getScore();
bestByParent.merge(parentId, score, Math::max);
}
List<String> top = bestByParent.entrySet().stream()
.sorted(Map.Entry.<String, Double>comparingByValue().reversed())
.limit(3) // answer from a few whole sections
.map(Map.Entry::getKey)
.toList();
return jdbc.sql("SELECT id, source, heading, body FROM section WHERE id IN (:ids)")
.param("ids", top)
.query(Section.class)
.list()
.stream()
.sorted(Comparator.comparingInt(s -> top.indexOf(s.id()))) // SQL does not keep our ranking
.toList();
}
Ingestion can parse Markdown into sections and store both granular and parent bodies:
public void ingest(Resource markdown, String source) {
MarkdownDocumentReaderConfig config = MarkdownDocumentReaderConfig.builder()
.withAdditionalMetadata("source", source)
.build();
List<Document> sections = new CarryHeadingForward().apply(new MarkdownDocumentReader(markdown, config).get());
for (Document s : sections) { // parents: stored whole, never embedded
jdbc.sql("INSERT INTO section (id, source, heading, body) VALUES (?, ?, ?, ?)")
.params(s.getId(), source, s.getMetadata().get("title"), s.getText())
.update();
}
List<Document> children = splitter.apply(sections); // each child: parent_document_id = its section
children.forEach(child -> child.setContentFormatter(formatter));
vectorStore.add(guard.apply(children)); // fail the job rather than truncate silently
}
Search stays sharp on small embeddings; generation receives wider context without stuffing the vector with tokens the embedder would ignore.
Mean pooling and attention masks in practice
After the ONNX forward pass, each surviving token has a hidden vector. Mean pooling sums those vectors and divides by the count of unmasked positions. Padding tokens contribute nothing when the mask is correct; truncated tokens contribute nothing because they never existed in the forward pass. People sometimes assume “the model read the Document object.” It read the tokenizer output. If your logging prints full chunk text beside a vector, you are looking at storage reality and search reality at once—and they disagree whenever length exceeds the window.
Similarity search then ranks by distance or cosine similarity against that pooled vector. A query about a refund exception buried in the second half of an 800-token policy chunk competes poorly against chunks that mention refunds up front. Authors did not write bad policies; the index refused to see the exception clause.
Advisor pipelines and citation hygiene
Chat clients that wrap retrieval advisors often concatenate document contents into a single context string. Unless metadata map keys survive into the response object or into deliberate citation markers in the prompt template, the model may answer correctly while the application cannot highlight the handbook section. Treat citation fields as part of the retrieval contract in tests: after each answer, assert that every used chunk id resolves to source, chunk_index, and heading when present.
Custom advisors can inject footnotes or JSON blocks the UI parses. Default demos skip that work, which is why so many internal RAG tools ship with “answers but no links.”
Overlap economics beyond the formula
The ceiling formula for chunk count under overlap is a planning tool, not a quality guarantee. Overlap helps when a sentence straddles a boundary; it cannot resurrect tokens the embedder drops. Paying for overlap while leaving max_length at 128 spends money to duplicate invisible tails. Fix window fit first, then add small overlap if boundary bleeding remains in evaluation misses.
Structure-first splitting details
Recursive splitters that try headings, then paragraphs, then sentences, then raw characters degrade gracefully. Heading-aware splits keep “## Refunds” with its bullets. Paragraph splits keep argument cohesion. Character splits are a last resort that should fail ingestion tests if they dominate a corpus—signaling that documents lack structure worth indexing as prose. For API reference Markdown, consider splitting on ## endpoints so each method is its own vector.
Embed-small/return-big retrieval path
At query time, similarity search returns small section ids. A join table or parent id in metadata loads the wider section body for the prompt. That pattern lets MiniLM specialize on tight semantic units while the LLM sees surrounding explanation. Implementation cost is a second store or a parent payload field—cheaper than pretending a 128-token window understood a full page.
Evaluation culture
Golden sets should include questions whose answers sit at the beginning, middle, and end of long sections. If only head-located answers pass, truncation is still biting. Include at least one table-oriented question and one code-oriented question if those formats appear in the corpus. Tag slow tests that download ONNX weights so CI can skip them on default PR runs while nightly builds still enforce vector sensitivity checks.
Configuration and best practices
Align splitter token rules with the embedder’s tokenizer, keep chunk bodies inside the embedding window, exclude noisy metadata from embeddings while retaining it for citations, prefer structural splits, use overlap sparingly, and special-case tables and code. Re-run a fixed evaluation suite after every change.
Testing strategy
Automated tests should lock the properties that demos usually skip.
Prove the end of a chunk changes its vector
@Test
@Tag("slow") // downloads the ONNX model on the first run
void theEndOfAChunkReachesTheVector() throws Exception {
TransformersEmbeddingModel model = new TransformersEmbeddingModel();
model.afterPropertiesSet();
String chunk = Files.readString(Path.of("src/test/resources/chunks/longest-chunk.txt"));
float[] original = model.embed(chunk);
float[] extended = model.embed(chunk + " The refund window is thirty days.");
assertThat(cosine(original, extended))
.as("identical vectors mean the appended sentence was truncated away")
.isLessThan(0.9999);
}
static double cosine(float[] a, float[] b) {
double dot = 0, na = 0, nb = 0;
for (int i = 0; i < a.length; i++) {
dot += a[i] * b[i];
na += a[i] * a[i];
nb += b[i] * b[i];
}
return dot / (Math.sqrt(na) * Math.sqrt(nb));
}
If appending distinctive text beyond the window does not move the vector, truncation is confirmed.
Prove recall does not drop when the chunker changes
@Test
void recallAtFiveHoldsTheFloor() throws IOException {
List<RetrievalCase> cases = RetrievalEval.load(Path.of("src/test/resources/retrieval-cases.tsv"));
double recall = new RetrievalEval(vectorStore).recallAt(5, cases);
assertThat(recall).isGreaterThanOrEqualTo(RECALL_AT_5_FLOOR); // recorded from the current chunker; only ever raised
}
Keep a floor on recall@K across splitter experiments so “improvements” that destroy known answers cannot merge unnoticed.
Verify every indexed chunk can be cited
@Test
void everyIndexedChunkCarriesItsSourceAndHeading() {
List<Document> sample = vectorStore.similaritySearch(
SearchRequest.builder().query("refund").topK(50).build());
assertThat(sample).allSatisfy(d ->
assertThat(d.getMetadata()).containsKeys("source", "title", "parent_document_id"));
}
Sample similarity results and assert source and heading metadata survive. A silent citation drop is a product defect even when answer text looks right.
Resources
Spring AI documentation for splitters, embedding models, and advisors; the ONNX MiniLM model card and its tokenizer.json; DJL tokenizer behavior; and your own retrieval evaluation fixtures. The durable lesson is architectural: retrieval quality begins at chunking. Tuning topK cannot recover text the embedder never saw.
Why demos hide the mismatch
Tutorials celebrate three-line indexing because it runs on a laptop and returns some answer. They rarely print tokenizer configs or compare WordPiece length to splitter length. Production failures then look like “RAG is unreliable” when the index simply never embedded the answer span. Spending a day on evaluation harnesses before prompt poetry usually shortens the overall project.
Operational checklist
- Print the embedder
max_lengthand special-token overhead. - Measure median and p95 chunk token lengths in the embedder’s tokenizer.
- Fail ingestion when a chunk exceeds the safe window.
- Keep a golden question set with answer spans and citation expectations.
- After each splitter or metadata change, run golden recall and citation tests.
- Prefer embed-small/return-big when parent context matters for generation.
Teams that treat those steps as optional rediscover them during an executive demo when the handbook answer lives on page two of an 800-token chunk.
Closing
Chunking is the retrieval system’s information bottleneck. Vectors average what they are shown; they cannot invent unseen tails. Align token languages, fit the window, split on structure, handle tables, preserve citations, and prove behavior with tests. Then topK, thresholds, and rerankers finally operate on a candidate set that deserves them.
Tokenizer mismatch as a first-class bug class
Engineering cultures that treat “800 tokens” as a universal unit will mis-size chunks forever. Print both splitter encoding and embedder encoding in startup logs. Convert a sample of production chunks through both tokenizers and histogram lengths. If the embedder p95 exceeds max_length - specials, you are truncating in production even when average length looks fine.
Teams migrating from OpenAI embeddings to local MiniLM (or the reverse) must re-chunk, not only re-embed. A corpus sized for one window is wrong for the other.
Why topK and rerankers cannot compensate
Raising topK increases the chance of retrieving some partially relevant chunk, but each chunk still hides its own truncated tail. Prompt budgets explode: ten 800-token chunks are thousands of tokens of mostly never-searched text. Cross-encoders that score full chunk text only see candidates the vector stage already proposed; they cannot promote a chunk whose searchable prefix never matched. Thresholds near zero already accept almost everything vector search returns; tightening them discards candidates without restoring invisible spans.
The repair is at ingestion: fit the window, or change the model, or split until fit is guaranteed.
Metadata policy matrix
Decide per field: embed, retrieve-only, or drop. Paths and UUIDs usually pollute embeddings. Headings often help disambiguation and should sometimes be embedded and always retrieved. Booleans like deprecated may belong in filters rather than text. Document the matrix beside the formatter builder so future edits do not casually flip enums.
Citation contracts in Spring AI advisors
If the default Q&A advisor flattens documents to strings, wrap or replace it so each used chunk keeps source, chunk_index, and heading. UI footnotes should resolve to those fields. Add a test that fails when citations are empty despite non-empty context. Product trust depends on that more than on BLEU scores.
Structural splitting recipes
For handbook Markdown: split on ## then paragraphs. For Javadoc-ish comments: split on declaration boundaries. For logs: do not embed entire stacks as one chunk; split on timestamps or error ids. For tables: one row or one logical record per chunk, carrying column headers as prefix text. For code: prefer function granularity over arbitrary token windows so call-site questions retrieve whole functions.
Overlap as a surgical tool
Use overlap when evaluation shows boundary misses on sentences that straddle cuts. Five to fifteen percent of chunk size is a common starting band; measure storage and embed cost before jumping to fifty percent. Overlap never fixes truncation; it only duplicates visible prefixes if the window is already full.
Embed-small/return-big operationally
Store section_id, parent_id, and body_small / body_full (or join to a sections table). Search on body_small embeddings; prompt with body_full. Monitor cases where small bodies lack enough signal—then adjust splitter, not only topK.
Test suite that locks behavior
- Truncation sensitivity: mutate chunk suffix beyond window; assert vector change policy (should be unchanged if truncated).
- Recall floor: golden questions with spans in head/mid/tail; fail builds that drop mid/tail recall after chunker changes.
- Citation survival: sample retrieval; assert metadata keys present.
- Length guard: ingestion fails closed when embedder token count exceeds safe max.
Slow tests that download ONNX weights belong in nightly lanes; keep a stubbed unit path for PR CI.
Operational checklist (expanded)
Align token languages; enforce window fit; structure-first splits; sparse overlap; table/code specialists; metadata matrix; citation-preserving advisors; golden eval before and after changes; embed-small/return-big when generation needs context the embedder cannot see. Chunking remains the retrieval system’s bottleneck—own it explicitly.