Reducing Hallucinations in a Medical RAG Chatbot Pipeline

The project began with a straightforward goal.

You want to build a chatbot capable of answering questions based on medical research papers.

That should be simple enough, right?

It turned out not to be.

The initial build followed a fairly conventional RAG setup: ingest the documents, generate embeddings, store them in a vector database, pull back relevant chunks, and pass them to the LLM.

It functioned.

Just not reliably.

And in a medical context, "not reliably" is a serious flaw.

A chatbot that answers confidently but incorrectly is far more dangerous than one that admits, "I don't have enough information to answer this."

That realization pushed the retrieval pipeline into a phase of continuous refinement.

The First Problem: Hallucinations

The most pressing issue to tackle was hallucination.

Language models are remarkably skilled at generating answers that sound convincing, sometimes too convincing.

When the documents didn't contain the answer, the model would often fill the gap using its own internal knowledge instead of admitting uncertainty.

That behavior needed to change.

The chatbot's answers had to come strictly from the supplied research material, not from the model's imagination.

So the underlying philosophy shifted.

Rather than treating the LLM as an authority on facts, the retrieved documents became the actual source of truth.

The LLM's role was reduced to interpreting that retrieved context and shaping it into a coherent response.

And when the context fell short?

The system should decline to guess.

Starting With Basic RAG

The first version of the architecture looked simple:

This represents the standard RAG pattern.

A large document gets split into smaller chunks, each of which is embedded and stored inside a vector database.

Whenever a user submits a question, that question is embedded too.

The system then searches for chunks whose embeddings are semantically close.

Straightforward, in theory.

But a pattern quickly emerged.

Semantic closeness doesn't guarantee actual relevance.

Why Vector Search Wasn’t Enough

Picture a user asking:

"What are the effects of insulin resistance?"

Semantic search is good at grasping the overall intent behind that question.

That's useful.

However, medical texts are full of precise terminology.

Terms like:

• insulin resistance
• HbA1c
• hyperglycemia
• metformin
• glucose tolerance

These specific words carry real weight.

Sometimes what you need is a grasp of general meaning.

Other times you need the system to locate the exact term itself.

Why settle for only one capability?

The solution was to combine both.

Hybrid Search

This is where hybrid search became a core part of the design.

Rather than depending solely on vector-based retrieval, semantic search was paired with keyword-based search.

The logic behind it is fairly intuitive.

Semantic search essentially asks:

"What content shares a similar meaning?"

Keyword search instead asks:

"Where do the key terms actually appear?"

Each method has its own strengths.

And each has its own blind spots.

Combined, they cover a wider range of queries.

The resulting flow looked like this:

User Query
                        ↓
              ┌─────────┴─────────┐
              ↓                   ↓
        Vector Search       Keyword Search
              ↓                   ↓
              └─────────┬─────────┘
                        ↓
                  Combined Results
                        ↓
                     Reranker
                        ↓
                 Best Context
                        ↓
                       LLM
                        ↓
                      Answer

This shift changed how retrieval was approached going forward.

Yet another issue remained.

Retrieving Something Doesn’t Mean It’s the Best Thing

Suppose the hybrid search step returns 20 chunks.

That sounds promising.

But is every one of those chunks genuinely useful?

Not necessarily.

Some chunks might be highly on-topic.

Others might simply share overlapping vocabulary.

Still others might be tangentially related without answering the actual question.

Feeding all of it directly to the LLM isn't an ideal fix.

More retrieved context doesn't automatically translate into better answers.

In fact, it can hurt the outcome.

It adds more tokens, more irrelevant noise, and more latency.

So one more stage was introduced.

Reranking.

Why Reranking Made a Difference

The retriever's role can be summarized as:

Surface possible candidates.

The reranker's role is different:

Determine which of those candidates are truly relevant.

So instead of the simple chain:

Query → Search → LLM

the pipeline evolved into:

Query
 ↓
Hybrid Search
 ↓
20 Candidate Chunks
 ↓
Reranker
 ↓
Top Relevant Chunks
 ↓
LLM

That separation of responsibilities matters.

The first retrieval stage can prioritize recall, casting a wide net.

The reranking stage can then focus specifically on relevance and precision.

For a medical research chatbot, this distinction proved especially valuable, since a chunk that merely contains the right terminology isn't always the chunk that actually answers the user's question.

The Piece That Mattered Most: Refuse to Fabricate

Beyond retrieval and reranking, strict guardrails were added at the generation step.

The core instruction boiled down to this:

Use the provided context to answer.

Do not invent information.If the context doesn't contain enough information,
say that there isn't enough information available.

That looks almost too simple to matter.

Yet it noticeably reshapes how the chatbot behaves.

Rather than pressuring the model to produce an answer no matter what, this approach gives it a way out when the material simply isn't there.

That escape hatch turns out to be essential.

Sometimes the honest response is something like:

"I couldn't find enough information in the provided research material."

Not every question deserves a confident answer.

So Is Hallucination Fully Eliminated?

Not really.

This became clear during the process of building the system.

RAG genuinely reduces hallucinations and keeps responses more tightly anchored to real sources.

But claiming zero hallucinations would be overstating things.

Plenty of failure points remain.

The retriever might pull the wrong chunk.

Chunking itself might strip away context that mattered.

The reranker might rank things poorly.

The underlying documents might be missing information to begin with.

And even when everything upstream works, the LLM can still misread what it retrieved.

So the real objective isn't:

"Build a chatbot that's never wrong."

It's closer to:

"Build a system with fewer chances to go wrong, and one that recognizes the limits of what it actually knows."

That's a far more achievable target.

Chunk Design Turned Out to Be Critical

One realization that stood out: splitting documents into chunks isn't a throwaway preprocessing step you configure once.

Chunks that are too large drag in unrelated content.

Chunks that are too small can strip away the surrounding context a statement depends on.

It helps to treat each chunk as a self-contained unit of knowledge rather than an arbitrary slice of text.

Well-designed chunks lead directly to stronger retrieval.

And stronger retrieval tends to produce better final answers.

Speeding Up the Pipeline

Correctness was only half the challenge; response time was the other.

A single RAG request can trigger several distinct operations:

User Query
   ↓
Embedding
   ↓
Vector Search
   ↓
Keyword Search
   ↓
Merge Results
   ↓
Reranking
   ↓
LLM

Running all of these strictly one after another slows everything down.

To address that, independent retrieval steps were converted to run asynchronously wherever possible.

The revised flow looked roughly like this:

User Query
                     ↓
              ┌──────┴──────┐
              ↓             ↓
        Vector Search   Keyword Search
              ↓             ↓
              └──────┬──────┘
                     ↓
                  Rerank
                     ↓
                    LLM

This cut down on idle waiting between steps that didn't actually depend on each other.

Accuracy alone isn't the whole story for a good RAG system.

Users don't want to sit around waiting for a response.

The Resulting Architecture

After several rounds of refinement, the pipeline settled into something like this:

Medical Research Documents
                         ↓
                  Document Processing
                         ↓
                      Chunking
                         ↓
                     Embeddings
                         ↓
                   Vector Database
                         ↓
                      User Query
                         ↓
              ┌──────────┴──────────┐
              ↓                     ↓
       Semantic Search        Keyword Search
              ↓                     ↓
              └──────────┬──────────┘
                         ↓
                    Result Fusion
                         ↓
                      Reranker
                         ↓
                Relevant Context
                         ↓
                 Grounded Prompt
                         ↓
                       LLM
                         ↓
                 Final Response

Each component in that diagram has a distinct responsibility.

That separation became one of the more valuable lessons from the project.

The vector database isn't there to answer questions.

The retriever isn't there to generate responses.

The LLM isn't there to know everything by default.

Every piece handles one job.

And ideally, it handles that job well.

Lessons From the Build

The main takeaway from this whole exercise is that RAG amounts to a lot more than pairing an LLM with a vector database. There are several moving parts, and each one deserves attention.

Retrieval Quality Is Non-Negotiable

Even the strongest language model can't compensate for poor context. Feed it weak retrieval results and you'll get weak answers back. It's a straightforward case of garbage in, garbage out.

Hybrid Search Earns Its Keep

Semantic search excels at capturing meaning and intent. Keyword search excels when precise terminology matters. Since medical research is full of exact terms and specific phrasing, combining both approaches turned out to be the right call.

Reranking Deserves More Credit

Pulling in twenty candidate results is one challenge. Narrowing those down to the five best ones is a separate challenge entirely. Reranking sits between those two steps and bridges the gap.

Bigger Context Windows Don't Guarantee Better Answers

There was an assumption early on that pulling in more retrieved content would naturally improve results. That assumption didn't hold up. In practice, five tightly relevant chunks often outperformed twenty mediocre ones.

Admitting Uncertainty Is a Strength, Not a Weakness

This might be the most important lesson of all. A trustworthy system shouldn't feel obligated to produce an answer no matter what. When the relevant information simply isn't available, the system should be willing to say so.

Where This Goes From Here

Plenty of room remains for improvement. Areas worth exploring next include:

• Stronger retrieval evaluation methods
• Query rewriting
• Metadata filtering
• Improved reranking models
• Responses that include citations
• Confidence scoring and abstention logic
• Better observability into the retrieval process
• Automated evaluation datasets
• Additional caching and latency improvements

Building a proper evaluation pipeline is a particular priority, since manually reviewing a handful of chatbot outputs isn't a rigorous way to judge quality. The kinds of questions worth measuring include whether the correct information was retrieved in the first place, whether the generated answer actually traces back to that retrieved material, and how frequently the system fails to surface the right context at all. Those metrics matter far more than a subjective sense of whether a response "sounds right."

Wrapping Up

What began as a straightforward RAG chatbot turned into a much deeper lesson in how retrieval systems actually work. Conversation around AI applications tends to focus on the language model, but in a RAG setup, the retrieval pipeline is doing much of the heavy lifting behind the scenes.

A minimal setup might look like documents flowing into a vector database and then into an LLM. A more dependable version looks more like documents going through chunking, then hybrid search, then reranking, then arriving as grounded context before ever reaching the LLM. And even that pipeline still has room to grow.

That's ultimately what makes building RAG systems compelling: it isn't just about getting a language model to produce text. It's about making sure that text is grounded in the right information before the model ever speaks.

Note: this project is meant for technical research and experimentation purposes only. It is not a substitute for professional medical advice, diagnosis, or treatment.