Designing a Grounded RAG Pipeline: Chunking, Filtering and Streaming

General-purpose language models reason, write and explain remarkably well until you ask about a system they have never seen: your internal documentation, your domain workflows, the exact behavior of an enterprise application your team configures every day. That knowledge is not in the training data, and no prompt trick puts it there. Worse, the model rarely admits the gap; it produces a fluent, confident guess. Below is the foundation of a grounded retrieval-augmented generation (RAG) system: how documents are prepared, how chunks are built, how candidates are filtered, how the prompt is shaped and how answers reach the user, with the reasoning behind each decision so you can apply it to your own build.

Why grounding changes the model's job

The premise of RAG is easy to state. Rather than trusting the model to remember an answer, you hand it the answer to read. Your real documentation is split into searchable pieces; when a question arrives, the system locates the passages with the best chance of answering it and places them in the model's context. The task shifts from "recall this fact" to "read this passage and explain it well", which is precisely the kind of work language models do reliably.

The concept is simple, but getting it to work well is not. Retrieving the right pieces, keeping them coherent and answering fast enough that nobody stares at a spinner all involve decisions that are easy to underestimate. The system described here is deliberately the foundation stage: retrieve, then answer. More ambitious extensions, such as knowledge graphs or a tool-using agent, build on exactly these pieces, and they only work if the foundation is solid.

Markdown as the source of truth

All of the knowledge the system answers from lives in Markdown files. That choice is practical rather than aesthetic. Markdown carries enough structure to preserve meaning, including headings, hierarchy, lists and tables, yet it remains plain text that can be chunked and embedded without wrestling a heavier markup format.

The files are written the way people naturally write technical documentation: a heading per topic, subheadings for the details beneath it, tables where structure matters, and screenshots wherever a picture explains faster than prose. Nothing about the authoring style is bent to suit the pipeline. That is important, because documentation that must be written for the retriever tends not to get written at all.

Keeping screenshots out of the embedding path

Screenshots deserve their own design decision. In application documentation they are not decoration: a large share of how-to questions are really asking which button or menu to use, and prose alone handles that badly.

Instead of embedding images inside the documentation files, the images are hosted separately and each Markdown file references them by URL. The link is just text, so it travels through chunking, embedding and retrieval like any other text. When a retrieved chunk containing such a link ends up in an answer, the frontend reads the URL and loads the image at render time. The user sees the documentation as it was written, images included, while the ingestion and retrieval pipeline never processes image data at all.

The principle is worth naming: keep content as text through every stage that only needs text, and resolve richer media at the last possible moment, when something is about to be shown to a person.

Streaming the answer as it is generated

The same "last step" thinking applies to delivering answers. Once generation starts, there is no reason to make the user wait for the complete response. Tokens stream back over a persistent connection as the model produces them, so the answer builds up a few words at a time, much like someone typing an explanation. Combined with images that load asynchronously as their URLs appear in the stream, the experience feels like a conversation, even though under the hood it is very much a database query followed by generation.

Hierarchical chunking: respecting the document's shape

Before anything can be searched, the documentation has to be divided into units that can each be embedded and retrieved independently. This is the stage where many RAG systems quietly cut corners, and the consequences are easy to miss.

Why fixed-size slicing fails quietly

The naive approach is mechanical: pick a token count, cut the document into pieces of about that size, and move on. It is quick to build and wrong in a way that never raises an error. A fixed-size window has no notion of sentence boundaries, and even less of conceptual ones. It happily splits a numbered procedure down the middle, strands a table away from the heading that gives it meaning, or leaves a definition in one chunk while its example lands in the next.

Such chunks have the correct length but the wrong shape, and retrieval quality depends on shape. If the retrieved text does not contain a complete thought, careful prompting downstream cannot repair it. The model is reasoning over a fragment, and the answer reflects that.

Reading the heading tree instead

Hierarchical chunking starts from the observation that documentation already has structure, and that the structure is an asset. Well-written technical documents form a heading tree: major sections, subsections nested beneath them, and body text attached at each level. Rather than ignoring that tree, the chunker walks it:

• Each major section becomes a chunk of its own.
• Each subsection becomes its own chunk too, but one that records its parent section.
• If a subsection is short enough that it reads better alongside its parent, the two are merged into a single combined chunk instead of producing a tiny, context-poor fragment.
• Content that does not sit under any heading, such as introductions, stray paragraphs or notes, is still captured as its own chunk type rather than being dropped or forced awkwardly into a neighbor.

Metadata that makes later stages possible

Every chunk carries more than its text:

• A breadcrumb: the chain of ancestor headings. A chunk about a single configuration option still knows it belongs to a broader workflow, even when retrieved on its own.
• A type label: telling apart top-level sections, nested details, merged parent-and-child chunks and standalone notes.
• A stable identifier: linking the chunk to its precise origin in the source file.

This metadata is not decoration. It is what makes reranking, reassembly of split sections and coherent, attributable answers possible later on. A flat list of equally sized text blobs cannot support any of that; a chunk that knows its place in the tree can.

The underlying trade-off is to follow the structure the document already has rather than overlaying an arbitrary one. It pays off quickly: chunks read like complete thoughts because they are complete thoughts, scoped the way the documentation's writer scoped them.

An adaptive second pass for the embedding token limit

Hierarchical chunking gets the shape right, but it is unaware of a hard constraint underneath: the embedding model accepts only a limited number of tokens per input. A document's structure does not care about that limit. A long FAQ, a sprawling reference table or a subsection that simply runs long can be a perfectly coherent chunk and still exceed what the model will accept.

This failure is dangerously quiet. Depending on the client, an oversized chunk is either rejected or silently truncated before embedding, and in both cases content disappears without anyone noticing at ingestion time. Truncation is the more insidious outcome, because the chunk still exists and still gets retrieved; its vector simply no longer represents the text you think it does.

The fix is a second pass downstream of the hierarchical one. Every chunk is measured against a safety threshold placed well under the model's true maximum, so there is a buffer rather than a cliff edge. Token counts vary between tokenizers and a margin absorbs that uncertainty. Chunks under the threshold pass through unchanged. Chunks above it are split further, and each resulting piece is:

• explicitly labeled as one part of a larger whole, so later stages can find its siblings
• given a small overlap of text from the adjacent piece, so nobody reading it, human or model, hits an abrupt, context-free cut in the middle of a thought

Where exactly to cut, and why naive splitting is not good enough even here, is a substantial design topic of its own. For the foundation, the essential point is that the hierarchical pass guarantees shape and the adaptive pass guarantees fit, so that good shape never costs you content.

Storing vectors with their metadata

Embedded chunks need a home built for one question: which of these many vectors are closest to this new one, answered quickly and at scale. Relational databases are not built around that query, so a purpose-built vector store takes the job.

The database runs in a container rather than being installed directly on the host. The reasons are practical: a containerized instance is easy to start, discard or move to another machine, and it brings no dependency baggage onto the host system.

Alongside each vector, the store keeps a complete metadata payload with the chunk's text, breadcrumb, type and source identifier. Each similarity hit therefore arrives with all the context required to use it at once, rather than as an anonymous vector. Pairing fast similarity search with rich metadata attached to each result is what allows filtering, reranking and sibling reassembly to work without a second lookup against some other source of truth.

The request lifecycle: from question to streamed answer

This is where the system does its real work, so it is worth describing precisely enough that the logic can be reused.

Separating submission from streaming

The API exposes two endpoints with different jobs. The first receives the question and immediately returns a conversation identifier. The second is a streaming endpoint that the frontend connects to with that identifier. Accepting a question and delivering an answer are distinct responsibilities, and separating them is what lets the response flow back incrementally, rather than holding a single request open until one monolithic reply is ready. It also gives the client a clean handle for reconnecting or correlating logs for a specific conversation.

Between those two endpoints, the following steps run in order.

Step 1: embed the question with the ingestion model

The user's text is embedded with exactly the same model used during ingestion. This is not a detail you can vary. Similarity comparisons are only meaningful when the query vector and the stored vectors live in the same vector space. If the embedding model ever changes, every stored chunk must be re-embedded, because vectors from two different models are not comparable even when they describe identical text.

Step 2: cast a wide net

The query vector is compared with the stored chunk vectors, and roughly the twelve closest matches come back. The measure is cosine similarity, which in essence measures the angle between two vectors: the smaller the angle, the more similar the meaning. This stage is intentionally generous. Its purpose is recall: making sure nothing relevant is excluded before the real selection begins.

Step 3: narrow the candidates through three gates

Three filters then run in sequence to reduce those dozen candidates to the few that matter:

1. A similarity floor. Candidates below a minimum score are removed. These are chunks that made the list only because nothing better existed, and keeping them would add noise.
2. A reranking pass. The survivors are rescored by a model that works differently from the first search. Instead of embedding the question and each chunk separately and comparing the vectors, it takes the question and one chunk as a joint input and judges whether that chunk genuinely answers the question. That weeds out a particular false positive: passages that sit near the query in vector space yet turn out not to hold the answer when read alongside it.
3. A strict floor and a hard cap. After reranking, a much stricter threshold trims what remains, and whatever survives is limited to a small, fixed maximum.

Each gate has one job: the first protects recall from noise, the second supplies precision, and the third enforces a hard limit on how much context reaches the model. What remains is the best available material, not merely the top of a long list.

The reason for this ordering is cost. The pair-reading reranker is far more accurate than vector comparison but also far more expensive, because it has to process each question-chunk pair jointly. Running it on a dozen pre-filtered candidates instead of the whole corpus gets its precision without its full price.

Step 4: restore split sections

Some surviving chunks are only parts of a section that the adaptive pass had to split. If the model receives, say, the second of three parts without any hint that the others exist, its answer rests on partial information. Therefore, before the prompt is assembled, every surviving chunk is checked for siblings, the other parts of the same original section. When they exist, they are fetched and placed in order, each labeled with its position. The model then reads the whole section rather than a slice of it. This is exactly where the part labels and stable identifiers attached at ingestion pay off.

Step 5: assemble the prompt

All retrieved chunks, together with their restored siblings, form the grounded context. The user's question travels alongside it, and a system prompt, described in the next section, tells the model what role it is playing and how to use the material.

Step 6: stream the answer back

Generated text flows back over the streaming endpoint the client connected to in the first place, delivered in small increments instead of one blocking response. The user watches the answer form in real time.

In summary: vectorize the query, retrieve broadly, filter through three gates, fill in missing parts, prompt the model and stream its output. No single stage is exotic. The quality comes from clean handoffs between stages and from giving each stage exactly one responsibility.

Designing the prompt: format, model and tone

By the time the prompt is assembled, the hard retrieval problems are solved: the right chunks have been found, filtered and reassembled. What remains is just as easy to get wrong: presenting that material so the model produces a good answer, not merely a technically correct one.

Writing the prompt in Markdown

The prompt itself is written in Markdown. Again the reason is practical. Models have seen vast amounts of Markdown in documentation, README files and technical writing, so instructions in a familiar format are parsed more reliably than instructions in a custom structure the model has to decode. Headings separate the prompt's sections, the retrieved context is clearly fenced off from the surrounding instructions, and any content reassembled from split parts carries a visible marker, so the model can tell "one continuous idea assembled from pieces" apart from ordinary retrieved text.

Choosing a smaller, faster model for synthesis

The model that writes the final answer is a smaller, faster one rather than the largest available. That seems counterintuitive until you look at the job it is actually doing. It is not asked to derive anything from scratch, remember rare details or cover for missing knowledge; the heavy lifting took place earlier, during retrieval and filtering. Its remaining job is to take carefully selected, neatly organized context and turn it into a clear, quick answer in the right voice.

That is a synthesis task. When the context is already grounded, a compact model comes close to a far larger one in answer quality while responding faster and costing much less. A larger model's reasoning capacity is worth paying for when the model must work something out. It is much less valuable when the answer is already in front of it and the task is explaining it well. The trade-off does depend on retrieval quality: the weaker the context, the more a larger model's ability to cope with ambiguity matters, which is one more reason to invest in the filtering stages.

A deliberate prompt anatomy

The system prompt follows a fixed structure rather than an improvised one:

• Role. It opens by telling the model what kind of assistant it is and which domain it serves, so tone and assumptions are calibrated from the first line.
• Task. It states the job explicitly: read the retrieved context and answer strictly from it, without reaching into general knowledge or guesswork, even when an answer seems obvious.
• Confidence and gaps. A short preamble sets how confident or hedged answers should sound and what to do when the context genuinely does not contain the answer: say so plainly instead of inventing something.
• Tone and style. Formality, typical answer length and whether to lead with the direct answer or build up to it are all specified rather than left implicit.

Nothing is left to the model's default idea of what a helpful assistant sounds like. Every behavior is written down, much like onboarding a new teammate by explaining the team's communication norms before assigning any work.

The payoff is an assistant that answers confidently when it has grounds, admits plainly when it does not, and keeps a consistent voice across every conversation. That consistency does not come from the model; it comes from the prompt around it.

Routing by intent: regular chunks and screen chunks

Questions that look alike can ask for very different things. A question such as "How is pricing calculated in this workflow?" is conceptual and wants an explanation. One like "Which field on this screen takes the price?" is navigational and wants a specific field, button or area of the interface. Both may pull chunks from the same area of the documentation, yet the ideal responses have almost nothing in common. Treating them the same leads to a conceptual lecture for someone who only needed a location, or, worse, to describing an interface element in the abstract when the user needed its exact location.

Separating the knowledge at ingestion

The distinction is introduced at the chunk level, not only at answer time. Alongside regular prose documentation chunks, a second category holds content that describes screens and their fields: what each field is called, what it does and where it sits relative to the other fields on the same screen. These are not tagged after the fact. The classification happens as content is ingested, so by the time a question arrives there are two clearly separated pools of knowledge rather than one undifferentiated pile.

Choosing the answering posture at query time

When a question comes in, it is evaluated for which kind of chunk it is really asking for: conceptual documentation or specific screen and field detail. That classification influences which chunks are prioritized during retrieval and which answer prompt the model receives:

• A screen-oriented question gets a prompt tuned for precision about interface elements: literal, specific and focused on exactly where and what.
• A conceptual question gets a prompt tuned for explanation: broader and more willing to connect ideas.

The retrieval pipeline and the generating model are the same in both cases; only the answering posture changes, chosen according to what the question needs instead of pushing every answer through one generic template.

This matters more than it sounds. An assistant with a single answering style ends up sounding generic, however good its retrieval. Recognizing intent, not just topic, is what makes the assistant come across as paying attention to what was actually asked. If you adopt this pattern, keep a fallback for ambiguous questions, for example defaulting to the conceptual posture and including any strongly matching screen chunks, so that a misclassification degrades gracefully instead of producing a wrong answer style.

Keeping memory under control

Careful retrieval and generation are worthless if the service gradually exhausts its memory. A process that embeds text, keeps chunks resident, assembles prompts and streams output, repeatedly and sometimes concurrently, accumulates memory pressure quietly unless something actively manages it. Objects needed only for one request tend to linger longer than they should when nobody cleans up after them.

So cleanup is not left to chance. At natural boundaries, such as the end of a request or the completion of an ingestion batch, memory is explicitly reclaimed instead of trusting it to clear eventually. In practice that means dropping references to large intermediate objects, clearing per-request caches and, for locally hosted models, releasing any memory the runtime holds on to.

It is unglamorous work that never appears in a demo and is only noticed when it is missing: a service whose performance erodes over a long uptime rather than responding on its thousandth query as quickly as on its first. Getting it right is less about clever engineering than about discipline, treating memory as something every stage manages rather than something the runtime will surely handle. Watching memory over a long soak test, not just a short benchmark, is the easiest way to confirm the discipline is working.

Key takeaways

The foundation described here is a complete, grounded RAG pipeline: structure-aware chunking, token-safe embedding, multi-stage filtering, reassembly of split sections, a disciplined prompt that keeps the model honest about what it knows, and real-time delivery. The decisions that carry the most weight are these:

• Keep documents in a structured plain-text format and resolve images and other media only at render time.
• Chunk along the heading tree, and attach breadcrumbs, type labels and stable identifiers to every chunk.
• Add a second pass that enforces the embedding model's token limit with a safety margin, labeled parts and small overlaps.
• Always embed queries with the same model used at ingestion, and re-embed everything when that model changes.
• Retrieve generously, then narrow with a similarity floor, a pair-reading reranker and a strict final cutoff.
• Reassemble split sections before prompting so the model never reasons over an unmarked fragment.
• Let a smaller, faster model do the synthesis once retrieval has done the hard work, and specify its role, task, gap handling and tone explicitly.
• Classify intent, not just topic, and give each kind of question its own answering posture.
• Reclaim memory at request and batch boundaries so the service stays as fast after hours of uptime as it was at startup.

Each of these choices is modest on its own. Together they produce a system whose answers are grounded, coherent and fast, and they give you a stable base for more ambitious retrieval techniques later.