Fully local PDF RAG with LangChain, Ollama, and Chroma

Build a fully local RAG pipeline for PDFs with LangChain, Ollama, and ChromaDB — chat with documents without cloud APIs or data leaving the machine.

Why local matters

Cloud LLMs are convenient, but every query ships text across someone else’s network. Legal contracts, medical records, and internal reports often cannot accept that tradeoff. A local stack removes the leak while keeping natural-language Q&A over your own files.

What you get

• Ingest PDF files into a persistent vector store
• Answer natural-language questions grounded in those documents
• Run offline at inference time — no API keys required once models are pulled

Stack: LangChain for orchestration, Ollama for local chat and embeddings, ChromaDB as the vector store.

Architecture

Ingest path: PDF bytes enter a PyPDF-based loader, get chunked, then Ollama embeds each chunk into Chroma.

Ask path: a question hits the retriever, matching chunks become context, and the local Ollama chat model writes the answer.

Two phases keep costs honest:

1. Ingestion — once (or when documents change): parse, chunk, embed, persist
2. Retrieval — every query: top chunks become context for the LLM

You pay embedding cost when content changes; interactive queries stay fast.

Prerequisites

Install Ollama and pull the models used here:

ollama pull embeddinggemma:latest   # embedding model
ollama pull gemma4:latest           # chat model

gemma4 is instruction-tuned. For a lighter footprint try gemma2:2b; for stronger reasoning use gemma4:27b — update the config strings accordingly.

Project setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Key packages from requirements:

langchain
langchain-chroma
langchain-community
langchain-ollama
langchain-text-splitters
pypdf
chromadb
python-dotenv

Override defaults via environment without editing code:

OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=gemma4:latest
OLLAMA_EMBED_MODEL=embeddinggemma:latest

Ingestion pipeline (src/ingest.py)

Load documents

def load_documents(pdf_path: str | Path) -> list:
    path = Path(pdf_path)
    loader = (
        PyPDFLoader(str(path))
        if path.is_file()
        else PyPDFDirectoryLoader(str(path))
    )
    return loader.load()py

One helper covers a single file or a directory. LangChain loaders always return a uniform Document list either way.

Split into chunks

def split_documents(documents: list) -> list:
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
    )
    return splitter.split_documents(documents)

RecursiveCharacterTextSplitter prefers paragraph, then sentence, then word breaks so semantic boundaries survive. A 200-character overlap keeps a sentence that straddles a boundary present in both neighboring chunks, which reduces retrieval misses at edges.

Embed and persist

def build_vector_store(chunks: list) -> Chroma:
    embeddings = OllamaEmbeddings(model=EMBEDDING_MODEL, base_url=OLLAMA_BASE_URL)
    VECTOR_STORE_DIR.mkdir(parents=True, exist_ok=True)
    return Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        collection_name=COLLECTION_NAME,
        persist_directory=str(VECTOR_STORE_DIR),
    )

Ollama serves embeddings locally on port 11434. Chroma persists under vector_store/ (SQLite plus HNSW index files) so process restarts do not force a full re-embed.

Retrieval chain (src/retrieval.py)

LangChain LCEL composes steps with the | operator.

Prompt template

RAG_PROMPT = ChatPromptTemplate.from_messages([
    (
        "system",
        "You are an assistant that answers questions based on the provided context "
        "from PDF documents. Use only the information in the context to answer. "
        "If the answer is not in the context, say so clearly.\n\nContext:\n{context}"
    ),
    ("human", "{question}"),
])

The system message anchors answers to retrieved context — the core RAG guardrail against free-form hallucination from pretraining alone.

Build the chain

def build_rag_chain(k: int = 4):
    store = load_vector_store()
    retriever = store.as_retriever(search_kwargs={"k": k})
    llm = ChatOllama(model=MODEL_NAME, base_url=OLLAMA_BASE_URL)
    return (
        {"context": retriever | _format_docs, "question": RunnablePassthrough()}
        | RAG_PROMPT
        | llm
        | StrOutputParser()
    )

Left to right: the retriever picks the k most similar chunks via embedding similarity, helpers format those chunks into one context string, the prompt template merges context with the user question, and ChatOllama generates the answer.

Interactive ask loop

# Index a directory of PDFs (default: data/pdfs/)
python main.py ingest

A thin CLI (or HTTP wrapper) loads the chain once and serves questions until you quit. Keep k tunable so you can trade context size against latency.

Running end to end

# Index a specific file
python main.py ingest --path /path/to/contract.pdf
# Ask a one-off question
python main.py query "What are the payment terms?"
# Interactive session
python main.py chat

Point ingest at your PDF folder, confirm the vector store directory filled, then query. Everything — embeddings, vectors, generation — stays on localhost.

Takeaway

Local RAG is no longer exotic. Ollama supplies models, Chroma persists vectors, LangChain wires the graph. For private corpora, that trio is enough to ship a useful document assistant without a cloud bill or an outbound data path.