Why Bad Chunking Wrecks Vector Search, Even with Good Embeddings

When vector search returns nonsense, the database and the embedding model get blamed first. Both are usually innocent. In a TypeScript ingestion pipeline built on PDF parsing, Gemini embeddings and ChromaDB, the culprit turned out to be one line of character-counting logic in the chunker.

Everything looked fine at first. The code compiled and the vectors sat neatly in the store. Then a query about student registration rules, run against a university policy PDF, came back as pure garbage.

What the first chunker did

The initial mental model was embarrassingly simple:

1. Extract text from the PDF.
2. Cut it every 800 characters.
3. Embed each slice.
4. Push the vectors into ChromaDB.

Instead of tweaking the database settings, the useful move was to print the raw chunks and read them. A key sentence about de-registering students who hadn't paid their fees had been cut straight through the middle word:

• Chunk 0 ended with: "...the right to decline or d"
• Chunk 1 began with: "e-register students who have not completed fee payments."

"de-register" became a stray "d" in one chunk and "e-register" in the next. The broken word was ugly, but the loss of meaning was worse. Chunk 0 no longer contained the action. Chunk 1 no longer contained the subject or the condition that explains why the action happens.

So the vector database wasn't missing the relevant passage. The meaning had been destroyed before the embedding model ever saw it.

Overlap gives a second chance

The next version added a 100-character overlap, so each chunk re-captured the tail of the previous one instead of starting as an isolated 800-character block.

The common assumption here is that overlap keeps a sentence from being split. It doesn't. A fixed boundary can still land somewhere terrible. What overlap really does is give text near a bad cut another opportunity to appear next to its surrounding context.

Think of it as insurance, not prevention: overlap doesn't stop the cut, it just gives the meaning a second shot at surviving it.

Retrieval got better right away, but the design was still wrong at its core, because it reasoned in raw characters instead of language. It could still slice a token or a code snippet in half and leave you with something like function calculateT.

That meant changing the question the chunker asks:

• Old question: where do the next 800 characters end?
• New question: where is the most natural semantic boundary to cut this text?

The answer was recursive chunking. Try the largest structural separators first, such as sections and paragraphs, and fall back to smaller ones like lines, sentences and words only when a piece is still over the size limit. A rule worth pinning above the code:

// The Golden Rule of Chunking:
// Preserve meaning at scale; cut aggressively only when forced to.

Every stage inherits the last one

Recursive splitting is just an implementation detail. The bigger lesson is how representation cascades through an AI system:

Doc ➔ Extraction ➔ Chunking ➔ Embedding ➔ Storage ➔ Retrieval ➔ LLM ➔ Answer

Each node downstream is limited by whatever the node upstream hands it. If ingestion wrecks the context, no amount of prompt engineering or index tuning will bring it back. Garbage in, semantic garbage out.

Fixing one thing exposes two more

Once the chunker behaved, two new problems showed up right behind it.

• Structure: paragraphs are easy, but what about a financial table that runs across three pages, or a multi-column layout? Split naively, the column headers end up separated from the values underneath them, and the data stops meaning anything.
• The price of overlap: context isn't free. More overlap means more chunks, which means more embedding API calls, more storage and slower indexing. With 10,000 vectors you won't notice. At 1,000,000 it's a real engineering and cost bottleneck.

How the mental model changed

Before, chunking meant splitting text into pieces so it fits in storage. Now it means deciding which information has to stay bound together so downstream models can represent and retrieve it correctly.

Writing the code was the easy part. Finding the hidden assumption that broke everything was the hard part, and that's the habit worth keeping: build it, break it, work out why, and come away with a sharper model of the system.