A RAG Project Is Not One Giant Python File: A Folder Layout That Scales

A RAG demo takes an afternoon: load a PDF, split it, embed the pieces, drop them into a vector database, ask a question, read the answer. That is plenty for a weekend experiment. As a base for real work it falls apart fast, and the reason is rarely the model or even the vector store. The reason is structure.

Beginner projects tend to begin as a single app.py that holds file loading, chunking, embeddings, storage, retrieval, prompt templates, model calls, API routes, logging and configuration. Opening one file feels convenient until the project grows. A new document source arrives, a different chunking strategy needs a trial, ChromaDB gets swapped for Pinecone, the prompt changes, a FastAPI endpoint appears, retrieval quality turns out to be poor and someone wants tests. By then that one file hurts to touch.

A maintainable system gives each concern its own home. Ingestion loads data, chunking splits documents, embeddings turn text into vectors, the vector database stores them, retrieval finds useful context, prompts shape the request, the LLM layer makes model calls and the API exposes everything to users. That separation is what turns a demo into an engineering system.

One mental model: pipeline versus application

Think of it this way: the pipeline prepares knowledge, and the application uses it. Keep them apart.

The pipeline side covers loading documents, cleaning text, splitting content, creating embeddings and storing them. The application side covers receiving questions, retrieving chunks, building prompts, calling the model, returning answers through an API and logging behavior in production.

Mixed together, every change is risky. Trying a new chunk size shouldn't require editing an API route. Moving from one model provider to another shouldn't mean rewriting retrieval. Adding website ingestion shouldn't touch prompt templates.

A starting layout

Here is a layout that works as a first draft. It is not the only valid one, but each folder has a clear reason to exist.

rag-project/
|-- README.md
|-- requirements.txt
|-- .env
|-- .gitignore
|-- config.yaml
|-- main.py
|-- src/
|   |-- ingestion/
|   |   |-- __init__.py
|   |   `-- loader.py
|   |-- chunking/
|   |   |-- __init__.py
|   |   `-- chunker.py
|   |-- embeddings/
|   |   |-- __init__.py
|   |   `-- embedder.py
|   |-- vectordb/
|   |   |-- __init__.py
|   |   `-- vector_store.py
|   |-- retrieval/
|   |   |-- __init__.py
|   |   `-- retriever.py
|   |-- prompts/
|   |   |-- __init__.py
|   |   `-- prompt_templates.py
|   |-- llm/
|   |   |-- __init__.py
|   |   `-- llm_client.py
|   |-- api/
|   |   |-- __init__.py
|   |   `-- routes.py
|   `-- utils/
|       |-- __init__.py
|       `-- helpers.py
|-- tests/
|   `-- test_app.py
`-- logs/
    `-- app.log

README.md

The README is the front door. It should say what the system does, how to set it up and run it, and what each major folder is for. Beginners often postpone it, yet a good one helps you understand your own project a month later. At minimum include the purpose, setup steps, required environment variables, how to ingest documents, how to run the API, sample questions and known limitations. For a portfolio piece it matters even more, since a reviewer may skim it before deciding whether to run anything.

requirements.txt

This file lists the Python packages the project needs. A simple RAG service might include something like the following.

fastapi
uvicorn
python-dotenv
pydantic
langchain
chromadb
sentence-transformers
openai
pypdf

Your list depends on your stack. The rule is not to leave dependencies hidden on your own machine. Anyone who clones the project should be able to install everything with a single command.

pip install -r requirements.txt

That makes the project reproducible and easier to deploy and review.

.env and .gitignore

The .env file holds sensitive values such as API keys. A typical one looks like this.

OPENAI_API_KEY=your_key_here
VECTOR_DB_URL=your_vector_db_url

It must never reach GitHub, so your .gitignore should exclude it along with logs and bytecode.

.env
logs/
__pycache__/
*.pyc

Hardcoding keys in Python files might work locally but is unsafe. Keep secrets outside the code.

config.yaml

Non-secret settings belong in config.yaml, for example:

chunking:
  chunk_size: 800
  chunk_overlap: 120

retrieval:
  top_k: 5

models:
  embedding_model: text-embedding-3-small
  llm_model: gpt-4.1-mini

vector_db:
  provider: chromadb
  collection_name: company_docs

RAG systems need tuning, and you will want to change chunk size, overlap, how many results to retrieve, model name, temperature or vector database options. When those values are scattered across random files, tuning gets annoying. Settings should be easy to find and change.

The code folders

ingestion/

This folder reads from PDFs, websites, CSV files, APIs, databases, Markdown, Notion exports, Drive folders or whatever else you have. Its job is not to answer questions but to bring raw content in, in a clean and predictable format. A loader.py might expose functions such as load_pdf(path), load_website(url), load_csv(path) and load_database_table(connection, table_name), each returning documents in one standard shape, normally text plus metadata.

Metadata deserves more attention than it usually gets. A chunk of text is useful, but a chunk with its source name, page number, URL, document type, author and timestamp is far more useful, because that data powers filtering, citations, debugging and trust.

chunking/

Chunking looks trivial and has a huge effect on quality. Oversized chunks return broad sections full of irrelevant text, while tiny ones strip away context the model needs. The goal is to preserve meaning, so instead of cutting every 500 characters blindly, consider splitting by headings, paragraphs, sections, pages, code blocks, tables or semantic boundaries. A chunker.py turns full documents into pieces ready for embedding.

Isolating this logic pays off because it is one of the quickest levers for improving a RAG system. With chunking separate, new strategies can be tested without touching anything else.

embeddings/

Embeddings are numeric representations of text that let the system compare meaning: the question is embedded and matched against embedded chunks. An embedder.py should centralize loading the model, embedding single inputs and batches, handling rate limits, retrying failures and tracking which model version produced the vectors.

The separation matters because embedding models change. You might start with a local sentence-transformer, move to a hosted API later, and switch providers again for cost or quality. With one place for that logic, each change stays manageable.

vectordb/

This folder owns storage of embeddings and metadata. Typical options are FAISS for local experiments, ChromaDB for simple local or small production setups, Pinecone for managed cloud search, and Weaviate, Qdrant or Milvus for larger or more specialized needs.

A vector_store.py should hide database specifics from the rest of the project. Retrieval code shouldn't know the details of Pinecone or ChromaDB. It should call plain project-level methods such as add_documents(chunks), similarity_search(query, top_k), delete_collection(name) and get_document_by_id(id). If you ever replace the database, you change this layer and nothing else.

retrieval/

Here a question comes in and the most relevant chunks go out. Basic retrieval is plain similarity search, while richer versions add metadata filters, hybrid search, reranking, query rewriting or multi-query retrieval. It matters enormously, because the model can only answer well when it receives the right context, and weak retrieval means weak answers.

A retriever.py may embed the query, search the store, apply filters, rerank, drop duplicates and return clean context for the prompt. Separating it makes debugging far easier, since you can ask whether the model failed or retrieval sent the wrong context. That question has no clean answer when the two are tangled together.

prompts/

Prompts shouldn't be scattered through the codebase. A prompt_templates.py can hold a template like this one.

You are a helpful assistant answering questions using the provided context.

Use only the context below. If the answer is not in the context, say you do not know.

Context:
{context}

Question:
{question}

Answer:

Pulling prompts into their own file is a habit worth forming early. They change often as you experiment with instructions, output formats, citation styles, safety rules or refusal behavior. Buried inside routes or model calls, they become hard to version and test. Treat them as part of the design and give them a dedicated file.

llm/

All language-model calls belong in one place, whether the provider is OpenAI, Claude, Gemini, a local Ollama model or something else. The rest of the application shouldn't care. An llm_client.py can handle initialization, API calls, retries, timeouts, temperature, response parsing, token usage logging and provider-specific errors. Models change quickly: GPT today, Claude tomorrow, a smaller model for cheap queries and a stronger one for hard ones next month. Centralized calls make switching easy.

api/

This layer lets other people and applications use the system, commonly through FastAPI or Flask. It should receive and validate requests, call the retrieval and generation pipeline, return structured responses and handle errors cleanly. It should not contain the logic itself. A route should be thin: accept the request, call the right service, return the result. If one endpoint contains ingestion, chunking, embedding, retrieval, prompt building and model calls, the project is becoming unmaintainable. The API should be the doorway, not the whole house.

utils/

Small helpers shared across the project live here, such as config loading, log formatting, text cleaning, path validation, timing and common error handling. Be disciplined, because this folder easily turns into a dumping ground. A helper that clearly belongs to ingestion goes in ingestion, and one that belongs to retrieval goes in retrieval. Use utils only for what is truly shared.

tests/

Tests separate serious projects from fragile demos, and RAG systems are made of many small parts that can be tested one by one. You can check that PDFs load, that chunk sizes look right, that metadata survives, that embeddings are created, that retrieval returns relevant documents, that prompts include both context and question and that the API returns the expected shape.

Perfect coverage on day one isn't required, but you need enough to know when something breaks. Retrieval tests are especially valuable: keep a small set of sample documents and questions, define which source should come back for each, and rerun them after any change to chunking or embedding settings. That gives a basic quality signal before users find problems.

logs/

Production logs may flow into a monitoring tool, but a simple log file helps during development. Logs should answer what the user asked, which chunks came back, which sources were used, how long retrieval took and which model was called. When a user says the answer was wrong, you need to see what was actually retrieved and what prompt was sent.

Keeping main.py short

The entry point should stay boring. It starts the API server, loads configuration or calls a main pipeline function, and nothing more. A simple entry point is a sign that the real work sits in the right modules.

What this buys you

• Debugging: isolate the failure. If the answer is wrong, inspect retrieval, then chunking and embeddings. If the API fails, inspect the route layer.
• Testing: check each component alone, so verifying the PDF loader doesn't need the whole app running.
• Scaling: improve one layer without rewriting the rest, whether adding batch ingestion, background jobs, caching, reranking or a new vector database.
• Collaboration: one person improves ingestion while another works on prompts or routes.
• Deployment: configuration, secrets, dependencies and entry points are clear.

Above all it makes the project understandable, and a project that is easy to understand is easier to trust.

A rule for first projects

Use one folder per responsibility, and don't aim for a perfect architecture. Just avoid one giant file. Start with ingestion, chunking, embeddings, a vector database, retrieval, prompts, LLM calls, an API, tests and logs. That is enough for most beginner and portfolio projects. Extra layers such as reranking, evaluation, caching, authentication, background jobs, document versioning and dashboards can be added as needs appear, while the base idea never changes: keep the pipeline separate from the application. A small habit that helps along the way is to write down, in the README, which folder you would touch for each kind of change, because if the answer is ever unclear the structure has probably started to blur.

A demo shows an idea can work. A maintainable project shows the system can grow. Folders for their own sake aren't the point. Giving every responsibility a clear home is.