A Policy Q&A Chatbot That Cites Its Sources Instead of Guessing

A chatbot that answers questions about public policy is easy to build and hard to trust. Fluent text isn't the goal here. The goal is an answer a citizen can check against an official document. This project, an open-source RAG chatbot called policy-qa-system, is built around that single idea.

Public, but unreadable

Governments publish huge volumes of policy material: benefit rates, application requirements, eligibility rules. Technically all of it is online. Practically, very little is usable by a normal person. Someone who just wants to know what they're entitled to or which documents to bring ends up facing a 30-page text written for caseworkers.

What follows is predictable. People phone a helpline to ask for something already published, or they give up and guess. In large document-heavy public services, this gap shows up constantly. The information exists. What's missing is a way to put a question to it.

The system fills that gap with retrieval-augmented generation. People ask everyday questions in plain language and get back an answer grounded in a real source document, with the citation attached. It isn't a general chatbot improvising about policy. It only speaks from the documents it was given.

Retrieve, read, answer, cite

Each question goes through four steps:

1. Search the indexed policy documents for the passages most relevant to the question.
2. Read those passages together with the question.
3. Write an answer in everyday language instead of policy jargon.
4. Attach the exact source, so the answer can be verified and not merely believed.

Most chatbot projects skip the last step, and here it matters most. An answer about entitlements is only useful if somebody can confirm it. So every reply carries a reference, and when nothing relevant turns up, the system says it isn't sure instead of guessing.

A plain-language example

Take the question "How much is the weekly payment for two children?" The reply looks like this:

For 2 children: first child gets the standard rate, second child
gets the additional-child rate (combined weekly and yearly totals
shown together). An income-based charge may apply above a set
household income threshold.

Source: Official rates document, section on child payments

The figures depend on which country's documents are indexed, since nothing in the system is tied to one set of rules. It's a pattern for citizen Q&A, not a fixed dataset.

Tech stack

• Frontend: React 18, TypeScript and Tailwind CSS
• Backend: Node.js with Express
• AI and RAG: LangChain and LangGraph
• Vector store: PostgreSQL with pgvector
• Deployment: Docker

Nothing exotic. A team using this for real would swap the sample documents for its own official policy set, while retrieval, citation and plain-language generation stay as they are.

Architecture

The request path runs from the user's question through the React interface to the Express API, which hands it to the retrieval chain. Here's the layout:

User question
      │
      ▼
 React UI ──► Express API
      │
      ▼
 LangChain retrieval ──► PostgreSQL + pgvector (indexed policy docs)
      │
      ▼
 LLM answer generation (grounded in retrieved sections)
      │
      ▼
 Answer + citation + confidence score

The confidence score deserves as much attention as the answer. When retrieval finds nothing close enough to the question, the system reports that, and the model never gets the chance to fill the hole with something plausible.

The code that matters

First, the chain doesn't produce free text on its own. It must return a source alongside the answer:

const response = await chain.invoke({
  question,
  context: retrievedChunks,
});

return {
  answer: response.answer,
  sources: retrievedChunks.map(c => ({
    document: c.metadata.document,
    section: c.metadata.section,
  })),
  confidence: response.confidence,
};

Second, a confidence threshold decides between answering and admitting nothing was found:

if (topMatchScore < CONFIDENCE_THRESHOLD) {
  return {
    answer: "I couldn't find a reliable answer to that in the indexed documents.",
    sources: [],
  };
}

That check is tiny, yet it separates a chatbot people can rely on from one that merely sounds sure of itself.

What worked

• Making citations mandatory improved retrieval from the beginning. A vague match becomes obvious once you must show where it came from.
• Even a crude confidence threshold sharply reduced wrong answers delivered with confidence. It looked like a small feature and had one of the largest effects on trust.
• Keeping the document set swappable means the same system could serve very different policy areas just by re-indexing.

What was hard

• Simplifying without distorting. Policy wording is precise for a reason, and making it friendlier without losing a condition such as an income limit or an eligibility rule takes real care. It isn't fully solved.
• Knowing when to say "I don't know". A threshold set too low lets the system answer things it shouldn't, and set too high it refuses questions it could handle well. The practical fix was tuning by hand against a small collection of real questions.
• Maturity. This is a base to build on and not a deployable product. Before real users see it, it needs a proper ingestion pipeline, continuing review of sample answers and testing against edge cases.

Where it fits

It belongs to a small family of open-source AI projects built around LLM orchestration with LangChain and LangGraph: a document Q&A system, a code review agent and a tool turning voice notes into Agile user stories. The shared bias is toward answers that can be checked, not answers that merely sound right.

Try it

The repository is quick to run locally:

git clone https://github.com/Srameshgitnow/policy-qa-system.git
cd policy-qa-system
npm install
npm run dev

The README covers configuration, env variables and how to index your own documents. Gaps in the citation logic and ideas for the confidence threshold are welcome as issues or pull requests.