From Prototype to Production RAG: Hybrid Search, Agents and Evaluation
A reference architecture for production RAG in .NET: chunking, hybrid and graph retrieval, re-ranking, query planning, typed output and ongoing evaluation.
This article is published in English.
Plenty of RAG prototypes seem complete after five steps: load documents, compute embeddings, save vectors, fetch a few chunks, prompt the model. Proving the concept takes no more than that. Trouble begins later, when the corpus swells, queries stop being predictable, an answer needs several sources, or somebody asks for numbers showing retrieval is improving. This reference architecture, shown through a .NET and Blazor application, covers what production adds. It starts with ingestion and chunking, then moves to retrieval that mixes keyword, vector and graph signals, re-ranking, an agent that plans queries, structured output, and evaluation that never stops.
The first diagram contrasts a prototype with the retrieval, ranking, grounding and evaluation stages production needs.
Three ways to give a model your data
A model answers from its training data plus whatever you supply at query time. Internal reports, contracts, support tickets, policies and project documents only help when the application can locate the relevant parts and put them into the model's context. That's easy until the data outgrows one prompt, and there are three usual approaches.
The simplest is the context window: extract the text of a file, paste it into the prompt and ask. It suits small amounts of content and short-lived tasks, and it turns wasteful once the same large content is sent over and over or the app must search many documents.
Project or session retrieval is the second. The application keeps a smaller working set available for repeated semantic lookup, which fits an assistant tied to one project, workspace or active session.
Third comes retrieval-augmented generation, abbreviated RAG. The app builds an index ahead of time, then each question pulls back just the passages that matter instead of shipping the entire corpus to the model. Microsoft's RAG overview frames it identically: a retrieval layer provides grounding data when a query arrives, and the model never needs retraining.
The next diagram compares the three by scope, lifetime and how much searchable data each can handle.
What retrieval really involves
Most of the engineering sits behind the word "retrieval." Lexical search wins when the user knows the exact term: product codes, customer names and policy numbers respond well to keyword matching. Vector search handles a different need, finding passages that mean the same thing in other words. The mechanism is easy to state. An embedding model converts text to numeric vectors, the question goes through that same model, and the search layer returns the closest neighbors.
What gets turned into a vector is the harder decision. One embedding for a whole long document tends to be too coarse for precise lookups. Production pipelines split documents into smaller, self-contained chunks, add some overlap so context isn't lost at the seams, and carry source metadata so each retrieved chunk can be traced home.
So ingestion is a pipeline rather than a single API call. Each file is parsed, normalized, split, embedded and indexed, and depending on the design that pass may also pull out metadata, fields, entities and relationships.
The diagram below follows source files through text extraction, chunking, embeddings, metadata extraction and searchable indexes.
One choice pays off later: keep enough metadata that every answer can return its source. A production RAG experience should let a reader follow any statement to the document and chunk behind it.
Mixing retrieval strategies
Vector search is useful but not equally strong on every query. Semantic similarity helps when wording shifts. Exact keyword matching stays valuable for names, dates, codes, acronyms and domain jargon. Graph retrieval supplies yet another signal when the answer hinges on relationships between entities instead of similarity between passages. Retrieval is therefore best treated as a blend of strategies and not a single search call.
Keyword search contributes lexical precision, and vector search contributes semantic recall. Hybrid search runs ranked methods together and merges their result lists. Azure AI Search does the merge with Reciprocal Rank Fusion (RRF), combining the ranked outputs of parallel searches. Graph retrieval goes a step further: the system uses extracted entities and relationships to follow links across content, which helps with questions whose answer is scattered over several documents.
Afterward a semantic re-ranker can rescore the candidates against the query and lift the strongest to the top. That second stage matters, since merging ranked lists doesn't tell you which result best answers the request.
The following diagram shows keyword, vector and graph searches merging through RRF and re-ranking before chosen chunks reach the model.
Query planning
A fixed pipeline presumes that one search is enough, which many real questions contradict. Imagine a user asking how the old policy differs from the new one, and what that means for contractors. Hidden inside are several jobs: locate two documents, extract the relevant sections from both, weigh the evidence, then write a synthesis.
Agentic retrieval puts this planning inside the retrieval workflow. An agent examines the request, splits it into focused subqueries, uses conversation history for context, runs the searches in parallel, re-ranks the results and merges the evidence into one grounded response. The architectural difference is that retrieval stops being a single lookup that always behaves identically. The application decides how many searches are needed and what evidence must be gathered before generation starts.
The planner is illustrated below, splitting one complex question into parallel retrieval tasks and fusing the evidence into a cited answer.
Evaluation
A RAG system can produce a polished answer even when retrieval was weak, so occasional manual spot checks can't be the only quality process. Three separate questions help:
- Retrieval quality: did search return chunks relevant to the query?
- Groundedness: do those chunks support the generated answer?
- Relevance: does the final answer address what was asked?
They isolate different failures. Weak retrieval can't be fixed by rewriting the final answer, a grounded answer may still miss the point, and a relevant answer may contain unsupported claims. Microsoft Foundry offers evaluators for these RAG quality dimensions.
The next diagram shows how the three measure different failure points across a RAG response.
In production, evaluation runs continuously: scores are collected and an alert fires when they sink below the expected level. Quality then stops being a one-time test and becomes part of application observability.
Putting the pieces together
With the responsibilities separated, the system is easier to reason about. Three flows emerge:
- Offline flow: prepares the corpus by parsing sources, chunking and embedding them, attaching metadata or entity relationships, then writing everything to the retrieval indexes
- Online flow: serves a request, with a planner choosing what to search, retrieval spanning the relevant strategies, merged and re-ranked candidates, and the selected evidence going to the model
- Feedback loop: scores retrieval quality, groundedness and relevance, revealing how changes in ingestion, ranking, prompts or data affect users
Microsoft's RAG solution design and evaluation guide separates these into distinct design problems, which is a handy way to organize the build.
The end-to-end picture, with offline ingestion, online retrieval and generation, and continuous evaluation, is next.
That split also shows why production work outgrows what a prototype hints at. Somewhere in the system sit document parsing, indexes, vector and lexical search, graph links, ranking, model calls, citations, evaluation and a UI. Managed services or a platform can provide several of them, yet each responsibility still exists, just under someone else's roof.
A worked example with financial statements
To exercise the architecture in an application, financial statements make good source material. They combine long documents, exact figures and repeated terminology, and a wrong number is obvious right away.
The application has four layers. A Blazor Server front end uses Telerik UI for Blazor for chat, charts, grids and layout. It calls the Nuclia .NET SDK for retrieval, and a RAG service owns ingestion, hybrid and graph retrieval, re-ranking and evaluation.
The layers appear in this diagram: Blazor Server, Telerik UI, the Nuclia SDK and the RAG service.
The financial PDFs passed through ingestion so searchable content and structured fields could be extracted. That gave the data two checks: standard retrieval for natural-language questions, and structured output for values destined for the UI. A test asking for fiscal-year net sales returned the figure with its source attached. Financial answers need the citation as much as the sentence, because it points the user straight at the filing.
The app relies on the SDK's AskAsync flow, and what makes it handy is that a response can be more than text. A C# type named ChartAugmentedAnswer goes in with the request. The SDK converts it to a JSON schema, the retrieval agent follows that schema when answering, and the JSON it returns is deserialized into a typed C# object.
This diagram traces that path from a C# type to a schema, then retrieved evidence, then typed chart data.
That removes a brittle step. Nothing has to parse prose to guess which figures belong on a chart. Text and structured values come back in one object, and the chart component binds right to them.
Picking the right level
A small, short-lived source may be handled by the context window alone. A larger corpus with direct factual questions is served by classic RAG as a baseline: good chunking, vector plus keyword search, citations and evaluation. Multi-part, conversational questions or ones that depend on links between sources justify query planning and agentic retrieval. Graph retrieval earns its keep when entity relationships are part of the question, but it adds complexity, so availability alone is no reason to adopt it.
The same applies to managed platforms. They cut down the retrieval plumbing a team has to maintain, while composing services yourself buys more control over indexing, orchestration, ranking and integration. The right call depends on which layer the team wants to own.
One piece can't wait: evaluation. Once users rely on the answers, a repeatable way to check that retrieval and generation still behave as expected is mandatory.
A prototype demonstrates that retrieval can work. Production has to demonstrate something tougher: the right evidence keeps turning up as data, questions and users shift. That's achievable once responsibilities are split. Create a tidy ingestion pipeline, retain citations and source metadata, mix retrieval methods where one signal isn't enough, re-rank before generation, plan queries when a question needs several searches, and track retrieval quality, groundedness and relevance so changes show up as measurements and not hunches. A sensible build order follows: begin with the smallest retrieval pattern that fixes today's problem, and add the next layer only when the application proves it's needed. What results is simpler to explain, test and improve.