End-to-End RAG with Chroma, FastAPI, and Llama 3.2

A grounded RAG assistant end to end: PDF chunking, cross-encoder rerank, Chroma, Llama 3.2 on Ollama, FastAPI, Docker, pytest and GitHub Actions in CI.

Publisher: REACTAPP.TOPAdded to this site Sep 26, 20268 min read1747 words

This article is published in English.

Sources: I Built a Production-Style RAG Knowledge Assistant with Python, ChromaDB, FastAPI, and Llama 3.2

Production-shaped RAG: ingest to CI

A grounded, source-cited knowledge assistant — PDF ingest, semantic search, reranking, local Llama generation, FastAPI, Docker, and GitHub Actions — framed as a learning system rather than a toy notebook.

After a first MCP experiment, the next gap was retrieval-augmented generation with evaluation and packaging, not another “upload PDF in a Colab.”

Goals

Index documents into a persistent vector store; retrieve, rerank, and ground answers with citations; serve over HTTP; measure quality with simple evals and pytest; containerize and wire CI so regressions are loud.

Architecture

Indexing sketch:

Documents
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Database
   ↓
Semantic Retrieval
   ↓
Reranking
   ↓
LLM
   ↓
Grounded Answer + Sources

Document loading path:

PDF Documents
      ↓
Document Loading
      ↓
Text Chunking
      ↓
Embeddings
      ↓
ChromaDB

Question-answering path:

User Question
      ↓
Semantic Retrieval
      ↓
Cross-Encoder Reranking
      ↓
Prompt Construction
      ↓
Llama 3.2
      ↓
Grounded Answer
      ↓
Source Citations

1. Ingestion

Load PDFs into LangChain-style documents, preserving page metadata for later citations. Bad loaders here create silent citation bugs later.

2. Chunking

Chunk size: 1000 characters
Chunk overlap: 200 characters

Overlap keeps sentences that straddle boundaries from disappearing at retrieve time. Treat sizes as config, not magic numbers buried in a notebook cell.

3. Embeddings

sentence-transformers/all-MiniLM-L6-v2

A small sentence-transformer keeps the demo runnable on a laptop while staying good enough for English technical docs.

4. Chroma persistence

Document
   ↓
Chunks
   ↓
Embedding Vectors
   ↓
ChromaDB

Chunks become vectors on disk so restarts do not force a full re-embed. Version the collection name when chunking strategy changes.

5. Semantic retrieval

documents = vector_store.similarity_search(
    query=query,
    k=top_k
)

Take top-k neighbors as the first cut before spending cross-encoder compute.

6. Cross-encoder rerank

cross-encoder/ms-marco-MiniLM-L-6-v2

User Question
      ↓
Vector Search
      ↓
Candidate Documents
      ↓
CrossEncoder
      ↓
Reranked Documents

Bi-encoder recall plus cross-encoder precision is the standard two-stage pattern for quality-sensitive RAG.

7. Grounding the LLM

Prompt construction injects ranked snippets as numbered sources:

[Source 1]
Retrieved document content...

[Source 2]
Retrieved document content...[Source 3]
Retrieved document content...

Instruct the model to answer only from those blocks and to cite them. Refuse when context is empty.

8. Local Llama via Ollama

Generation stays on-box for privacy and cost. Pin the tag in config and health-check the Ollama process before accepting traffic.

9. FastAPI surface

GET /health
POST /query

Health for probes; query for the RAG path. Keep the service thin — call the same pipeline the CLI used.

Full query flow reminder:

Question
   ↓
Semantic Retrieval
   ↓
Retrieved Chunks
   ↓
CrossEncoder Reranking
   ↓
Grounded Prompt
   ↓
Llama 3.2
   ↓
Answer + Citations

10. Evaluation

Questions evaluated: 5
Questions passed: 5
Keyword Match Rate: 100.00%

Hit Rate@K
Recall@K
MRR
Answer Relevance
Faithfulness
Citation Correctness

Keyword checks plus retrieval metrics (hit rate, recall, MRR, answer relevance) catch silent regressions when embeddings or prompts change.

11. pytest

API health endpoint
Document chunking
Document retrieval

3 passed

Cover health, chunking, and retrieval smoke tests so CI fails fast.

12. Docker lesson

Image size surprises are common when models sneak into layers:

1.36 GB

1.36 GB

2.56 KB

Ignore caches and virtualenvs properly:

.venv/
__pycache__/
*.pyc
.pytest_cache/
.git/
.github/
.env
chroma_db/
data/sample/*.pdf
*.log

Prefer mounting model caches at runtime so rebuilds stay small.

13. GitHub Actions

Push / Pull Request
        ↓
GitHub Actions
        ↓
Checkout Repository
        ↓
Set Up Python
        ↓
Install Dependencies
        ↓
Run pytest

Push and pull request runs should execute lint, unit tests, and optionally a slim eval job with fixtures.

Stack, difficulty, next steps

Document Processing
        ↓
Chunking
        ↓
Embeddings
        ↓
Vector Storage
        ↓
Retrieval
        ↓
Reranking
        ↓
Prompt Construction
        ↓
LLM Generation
        ↓
Citation Handling
        ↓
Evaluation
        ↓
API
        ↓
Testing
        ↓
Deployment

Hard parts were less “call an LLM” and more chunk boundaries, rerank latency, citation formatting, and Docker layer bloat. Next upgrades: hybrid search, a larger labeled eval set, auth in front of FastAPI, and observability around retrieve latency.

Production-style means evaluation and packaging travel with the notebook ideas — not that one laptop replaces a platform team.

Track chunk size and overlap in config so experiments stay comparable across branches.

← All articles Send the brief