Self-Hosted RAG for Document Q&A: LangChain, pgvector and Swappable LLMs

Documents rarely answer their own questions. Policies, reports and contracts pile up as PDFs, Word files and scans, and almost nobody has time to read them end to end. Building a retrieval-augmented generation (RAG) system that can answer questions about uploaded files is a practical way to fix that, and the open-source project AI-DocumentIntelligence is a good case study in keeping such a system simple and trustworthy.

The gap this fills

The same situation shows up in every organization with a lot of paperwork. A caseworker needs one line out of a 40-page policy. An analyst needs a single figure from page 22 of a report. The file is technically available, yet finding anything in it takes forever.

Teams usually end up with two options. One is a closed-source document AI product with unclear pricing and fuzzy rules about what happens to the data. The other is carrying on with manual searching.

An open, self-hosted alternative offers a third path. The goal isn't another thin wrapper around somebody's API. It's something simple enough that a team handling sensitive documents can inspect it, swap parts of it and trust it.

How it works: upload, chunk, embed, ask

The application does four jobs:

1. It accepts PDF, DOCX or TXT files.
2. It splits each file into small, meaningful chunks.
3. It converts the chunks into embeddings and stores them in PostgreSQL through the pgvector extension.
4. It answers plain-language questions about the document using either OpenAI or Anthropic Claude, and a single setting switches between the two.

That last item matters more than it looks. Many tutorial RAG projects tie you to one model vendor. Real projects often need to change providers because of cost, compliance or availability, and nobody wants to rewrite the app when that happens. Here LLM_PROVIDER is a plain configuration value instead of something buried deep in the code.

The stack

The technology list is deliberately short:

• Frontend: React 18 with TypeScript
• Backend: Node.js and Express in TypeScript
• Orchestration: LangChain
• Vector store: PostgreSQL with pgvector
• Models: OpenAI or Anthropic Claude, your choice
• Tooling: Docker, ts-node and dotenv

Nothing in there asks a team to learn a new way of shipping software. Anyone who can already run a Node.js service next to a PostgreSQL database can run this.

Architecture

The overall layout is a React interface talking to an Express API, which hands uploads to a document processor:

┌────────────┐      ┌──────────────────┐      ┌────────────────────┐
 │  React UI   │ ───► │  Express API      │ ───► │  Document Processor │
 │ (upload+chat)│      │ (TypeScript)       │      │ (chunking / split)   │
 └────────────┘      └──────────────────┘      └────────────────────┘
                                │                          │
                                ▼                          ▼
                      ┌──────────────────┐      ┌────────────────────┐
                      │  LangChain Layer   │ ◄──► │ PostgreSQL + pgvector│
                      │ (OpenAI / Claude)    │      │  (embeddings store)   │
                      └──────────────────┘      └────────────────────┘

Chunking and embedding happen once, at upload time. After that, every question takes the same short route. The question is turned into an embedding, pgvector returns the closest chunks, those chunks travel to the selected model along with the question, and the reply comes back together with the chat history for that document.

There's nothing exotic in this design, and that's the point. The value comes from getting the plain parts right: picking the provider cleanly, splitting text sensibly and failing gracefully when a key or setting is missing.

The code that matters

Only a few snippets show the core ideas.

Switching providers comes down to reading an environment variable:

// backend/src/llm/provider.ts (simplified)
const provider = process.env.LLM_PROVIDER; // "openai" | "anthropic"

A small factory function then returns the right chat model depending on that value:

export function getChatModel() {
  if (provider === "anthropic") {
    return new ChatAnthropic({
      apiKey: process.env.ANTHROPIC_API_KEY,
      model: process.env.ANTHROPIC_MODEL,
    });
  }
  return new ChatOpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });
}

Chunking relies on LangChain's recursive splitter, configured with Q&A in mind more than raw storage:

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 150,
});
const chunks = await splitter.splitDocuments(rawDocs);

Overlap deserves more attention than it usually gets. With too little, an answer can be sliced in half across two chunks. With too much, you pay to store and search the same sentence three times. An overlap of 150 characters on 1000-character chunks gave the best results across a mix of PDF and DOCX test files.

Retrieval and generation also stay apart. Each stage can be tested alone, so you can confirm that retrieval finds the right chunks without burning API credits on generation. On a free-tier OpenAI account, that saving is considerable.

What worked and what hurt

Decisions that paid off

• A swappable provider from day one. Most of the retrieval logic was built and tested with free local embeddings before any paid API was touched, so costs stayed near zero during development.
• Keeping chunking and embedding apart from question answering. When an answer looks wrong, you can quickly tell whether retrieval or generation is to blame.
• Docker Compose for the whole stack (backend, frontend and Postgres). Anyone can clone the repo and have it running in minutes instead of fighting dependencies for an afternoon.

Things that were harder than expected

• Working without paid API credits. Missing access to a paid OpenAI or Claude account is a real constraint. The workaround was to validate retrieval with free local embedding models first and add the paid generation step later. It's sound advice for any RAG build on a tight budget.
• Choosing a chunk size. It's unglamorous and it's critical. No amount of prompt tuning rescues a bad split, and correcting it produced the biggest single improvement in the project.
• Making it run on someone else's machine. A laptop where everything works and a clean docker compose up on a stranger's computer are different things. A clear .env.example, explicit database name settings and a documented fix for port conflicts took longer than the RAG logic itself, but they decide whether anybody else can use the project.

Running it yourself

The repo is open source and starts locally with a few commands:

git clone https://github.com/Srameshgitnow/AI-DocumentIntelligence.git
cd AI-DocumentIntelligence
docker compose up --build

Environment variables, full setup steps and troubleshooting notes are in the project README. Bug reports, ideas about the provider-switching approach and pull requests all fit in the issue tracker.

The wider idea behind projects like this is to use LLM orchestration with LangChain or LangGraph on real workflow problems, such as policy Q&A for public services, code review agents or turning voice notes into Agile user stories, and to skip the polished demo that never meets real documents. Simple, swappable and easy to inspect usually beats clever.