Stopping Chatbot Hallucinations: RAG, Vector Search, Chunking and Guardrails

A support chatbot gets asked about the company's refund policy and invents one. The policy doesn't exist, the customer follows it, and the company ends up honoring it. This is the hallucination problem in its most expensive form, and it is a favorite system design interview topic because it joins AI infrastructure with classic architecture.

Why models make things up

A large language model is a probabilistic text generator. It doesn't know facts, it predicts the next most plausible token. When the answer isn't in its training data or its context, it produces something that sounds right but isn't. If training text contains many refund policies from other companies, the model will output one that fits the pattern, even if it's wrong for this company.

This isn't a bug so much as a consequence of how the technology works. Fine-tuning alone doesn't fix it either, because you can't bake every possible fact into the weights. The main defense is retrieval-augmented generation, or RAG: rather than asking the model to answer from its parameters, fetch relevant documents first and place them in the prompt.

The scenario for design is a customer-support chatbot answering questions about products, policies and troubleshooting. It answers confidently, yet sometimes claims a product has a feature it lacks or invents a policy. How do you keep its answers grounded in the real documentation?

Topics worth covering in a good answer include:

• why models hallucinate and why fine-tuning isn't enough
• the RAG flow of retrieving context, then generating
• vector databases and embedding-based semantic search
• chunking strategies for splitting documents
• prompt techniques that reduce hallucination
• managing the context window within token limits
• evaluation pipelines that monitor answer quality

How RAG reduces hallucination

With RAG, the system first retrieves relevant material from a knowledge base and inserts it into the prompt as context. Without it, the model guesses from its parameters. With it, the model reads the answer from supplied documents. It still generates tokens, but the context constrains it. If the documents say there are no refunds, the model won't invent a 30-day return window.

Vector databases and embeddings

How does the system find the right documents? Each document is converted into a vector embedding, a high-dimensional numeric representation of its meaning, and the query gets the same treatment. The vector database returns the documents whose embeddings sit closest to the query's.

Four components do the work:

• Embedding model. It maps text to vectors so that similar texts produce similar vectors. Options include OpenAI's text-embedding-3-small, which has 1536 dimensions, and open-source models such as BGE or E5.
• Vector database. It stores the embeddings and supports approximate nearest neighbor (ANN) search. Choices include managed Pinecone, plus Weaviate, Qdrant, pgvector (a PostgreSQL extension) and Milvus.
• ANN search. Comparing a query to every stored vector gets slow. Algorithms like HNSW (Hierarchical Navigable Small World) and IVF (Inverted File Index) find approximate neighbors far faster, in roughly logarithmic time. With 10 million or more vectors, exhaustive search can take seconds while ANN takes milliseconds.
• Reranking. The first pass returns about 20 candidates. A more accurate cross-encoder then reorders them by relevance, and only the top 3 reach the LLM. Fast but approximate retrieval followed by slow but precise reranking balances speed and accuracy.

Chunking

A 50-page policy document can't be retrieved whole, so it has to be split. Chunk size affects both retrieval accuracy and how well the context window is used.

• A chunk that's too small loses context. It might read "Items damaged during shipping are not" with the rest of the sentence in the next chunk, leaving the model unable to answer correctly.
• A chunk that's too large dilutes the useful signal with irrelevant text and fills the window with noise.
• Overlap protects against boundaries. A sliding window with 10 to 20 percent overlap keeps information from falling between chunks.

A sound default for documentation is to split by section heading, cap chunks at roughly 500 to 1000 tokens and overlap neighbors by 50 to 100 tokens. Each chunk should be self-contained, a complete thought.

Prompt design as a second line of defense

Even with retrieval, a model may ignore the context and hallucinate anyway, so the prompt must firmly tie it to the supplied material. The key elements are:

• A role. Something like "You are a customer support assistant" sets the persona.
• Grounding. An instruction to answer only from the provided context keeps the model from drawing on what it memorized.
• A fallback for uncertainty. Telling the model to say it can't answer when the context lacks the answer is the single most valuable line. Without it, the model guesses.
• Citations. Requiring document sources forces the model to attribute claims, which makes fabrication harder.
• A ban on outside knowledge. An explicit prohibition on information beyond the context reinforces the grounding rule.

Managing the context window

What if the relevant chunks don't fit, or a question has several parts that need many chunks? Modern models have big windows, such as 128K tokens for GPT-4o and up to 1M for Gemini, but stuffing them with marginal material lowers answer quality. Four tactics help:

1. Rank by relevance. Retrieve ten chunks, order them by score and add them from best to worst until the token budget runs out, dropping the weakest first.
2. Decompose the query. For a question like "Tell me about the refund policy and shipping times", split it into sub-queries, retrieve for each and answer both in one reply.
3. Summarize large topics. When a subject spans many chunks, have the model summarize them first and use that summary as context for the final answer.
4. Widen around a match. For questions on one section, pull the chunks before and after the hit, so the context is complete without including the whole document.

The complete architecture

Putting it together, the main decisions are these:

• Semantic chunking. Split at natural boundaries such as headings and paragraphs, with about 500-token chunks and 50 tokens of overlap, each self-contained.
• Two-stage retrieval. ANN search yields 20 candidates and a cross-encoder reranker keeps the best 3.
• Low temperature. A setting of about 0.2 on a 0 to 2 scale makes the model less inventive and less likely to fabricate.
• Guardrails. A content filter screens for personal data and policy violations, a citation check confirms the answer references the context, and a confidence score derived from retrieval similarity decides whether to answer or decline.
• An explicit "I don't know". When retrieval confidence falls below a threshold, the bot says it can't answer. This is the strongest single defense.
• A feedback loop. Thumbs up or down from users feeds an evaluation pipeline that tracks accuracy and hallucination rate over time.

What to monitor

1. Hallucination rate. The share of answers containing information absent from the supplied context, measured through automated evaluation or human review. A reasonable target is under 1 percent.
2. Retrieval precision@3. Of the top three chunks, how many truly relate to the query. A figure below 70 percent suggests the embedding model or chunking needs work.
3. Confidence distribution. The split of answers given with high, medium or low confidence. Many low-confidence cases mean the knowledge base has gaps.
4. Citation accuracy. Whether a cited document really supports the claim, checked automatically.
5. User feedback score. The average rating from thumbs up and down, a measure of overall satisfaction.
6. Context utilization. How much of the available token budget gets used. Persistently low use means retrieval returns too few chunks, while use near 100 percent means the budget needs raising.
7. Ingestion freshness. The delay between a document change and its appearance in search results. In dynamic knowledge bases, staleness yields outdated answers.

The four layers of defense

Hallucination isn't a sign that the AI is broken. It's a design problem. An unconstrained model always chooses plausibility over accuracy, because that is what it was trained to do. Four layers address it:

• Retrieval. Supply relevant documents so the model answers from them and not from memory.
• Prompting. Constrain the model to the given context, tell it to say it doesn't know and require citations.
• Guardrails. Filter outputs, check citations and compute confidence scores, rejecting anything that can't be verified against the context.
• Feedback. Track hallucination rate, user ratings and retrieval accuracy, and keep improving the knowledge base and pipeline.

The rule that matters most: a chatbot that confidently says "I don't know" is far better than one that confidently makes up an answer.