Layer by Layer: Designing a Real RAG Retrieval Pipeline for AI Agents

Replacing a toy retrieval agent with a real RAG pipeline is a good moment to slow down and layer the work. In a multi-agent enterprise platform, the RetrievalAgent initially ran on a deliberately simple local implementation. That was enough to prove the architecture, but production use calls for real documents, embeddings, vector storage and semantic search. The target flow runs from documents through ingestion, chunking and embeddings into ChromaDB, then on to semantic retrieval, the RetrievalAgent and the wider platform.

Instead of building it all in one go, the plan adds one layer at a time, with tests at every stage and the CI pipeline kept green. Steady increments, a healthy CI and a sound architecture are the goals.

The pieces of a RAG pipeline

A complete pipeline bundles several independent engineering concerns, which the diagram below lays out from source documents onward.

Source Documents
       ↓
Document Ingestion
       ↓
    Chunking
       ↓
   Embeddings
       ↓
 Vector Storage
       ↓
Semantic Search
       ↓
Retrieved Context
       ↓
Retrieval Agent

Building every component simultaneously is possible, but it makes failures hard to pin down. When retrieval quality is poor, where's the fault? It could be document parsing, chunking, embeddings, vector storage, query logic, metadata or the agent integration. Giving each layer a clear responsibility, a contract and a test boundary keeps those questions answerable. The working rhythm is build, test, push, validate in CI, keep it green, then continue.

Start with the data contract

Before any embeddings or ChromaDB, the first job is the data contract. The pipeline has to tell apart two concepts: SourceDocument and DocumentChunk.

A SourceDocument stands for the original knowledge source before it's split. Eventually it could be a PDF, a DOCX or a TXT file, internal documentation, financial reports, enterprise policies, technical manuals or knowledge-base content. Its conceptual shape looks like this:

SourceDocument
 ├── document_id
 ├── content
 ├── source
 └── metadata

Each original document therefore gets a stable identity within the retrieval architecture.

Large enterprise documents shouldn't normally be embedded as one huge block of text. They need to be broken into smaller, meaningful units, and that's the job of DocumentChunk:

DocumentChunk
 ├── chunk_id
 ├── document_id
 ├── content
 ├── source
 ├── chunk_index
 └── metadata

The relationship between the two is one-to-many:

SourceDocument
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Chunk 0       Chunk 1      Chunk 2
          │            │            │
          └────────────┼────────────┘
                       ↓
                 Same document_id
                  + source metadata

A single source may yield dozens of chunks, or eventually hundreds, but every chunk keeps a link to its originating document_id and source. That link becomes critical once semantic retrieval starts returning individual fragments of much larger documents.

Why traceability is built in from day one

Picture a 50-page financial report. The system may divide it into dozens of chunks, and a semantic search might return only three. The system must still be able to say which original document the information came from. Reconstructing that relationship after retrieval is fragile, so the relationship should be preserved at every step of the pipeline.

Original Document
       ↓
SourceDocument
       ↓
DocumentChunk
       ↓
Embedding
       ↓
Vector Record
       ↓
Semantic Search
       ↓
RetrievedDocument
       ↓
Original Source

Traceability is part of the data architecture from the start and not an afterthought.

Testing the contracts

With the models defined, automated tests cover both SourceDocument and DocumentChunk. They check that the models construct correctly and that key fields behave as expected, including document IDs, source details, content and chunk indexes.

It might look like a minor milestone, yet these models will sit under the entire retrieval pipeline:

Document Models
      ↓
   Chunking
      ↓
  Embeddings
      ↓
   ChromaDB
      ↓
  Retrieval
      ↓
    Agents

A broken contract at the front of that chain can affect everything downstream, so the data foundation gets protected before more infrastructure goes on top.

Another deliberate choice was keeping this stage free of external dependencies. No OpenAI calls and no ChromaDB calls happen here, and the tests only validate the document contracts. The fast CI suite therefore stays deterministic.

Code Change
     ↓
Local Fast Tests
     ↓
Commit
     ↓
Push to GitHub
     ↓
GitHub Actions
     ↓
Clean CI Environment
     ↓
Fast Unit Tests
     ↓
CI PASSED ✅

The fast suite grows while external integration tests stay in their own separate group. The rule behind it: every new layer enters the platform with automated protection.

The full roadmap

The contract for documents is just layer one. Here is the entire target architecture:

Enterprise Documents
        │
        ▼
     Ingestion
        │
        ▼
      Chunking
        │
        ▼
     Embeddings
        │
        ▼
     ChromaDB
   Vector Storage
        │
        ▼
 Semantic Retrieval
        │
        ▼
 Retrieved Context
        │
        ▼
  RetrievalAgent
        │
        ▼
Multi-Agent Platform

Every stage has a specific responsibility and should remain understandable and testable on its own.

Chunking comes next

The next component is the Chunking Service, which turns a SourceDocument into several DocumentChunk objects.

SourceDocument
      │
      ▼
ChunkingService
      │
      ├── DocumentChunk 0
      ├── DocumentChunk 1
      ├── DocumentChunk 2
      └── DocumentChunk N

Over time the service has to handle chunk size and overlap, carry metadata and source details through, assign each chunk an index and generate stable identifiers. One architectural call stands out: chunking stays out of the Retrieval Agent. The agent consumes retrieval capabilities and shouldn't also take on document preparation.

Embeddings and storage

After chunking is implemented and tested, the next layer generates vector embeddings.

DocumentChunk
      ↓
Embedding Service
      ↓
Embedding Model
      ↓
Vector Representation

Every chunk turns into a vector that encodes what it means. Later, those vectors get compared against an embedded user query to find related material.

DocumentChunk
      ↓
EmbeddingService
      ↓
Embedding Provider
      ↓
Vector

Swapping the embedding model later should barely disturb the rest of the RAG design.

After embeddings come to persistent vector storage. For local development, ChromaDB's persistent client can keep the indexed knowledge across restarts of the application.

DocumentChunk
      +
Embedding
      +
Metadata
      ↓
   ChromaDB
      ↓
Persistent Vector Collection

A stored record needs more than a vector. It also carries enough information to maintain context and traceability:

Vector Record
 ├── chunk_id
 ├── embedding
 ├── content
 ├── document_id
 ├── source
 ├── chunk_index
 └── metadata

That lets retrieval return meaningful document context and not just similar vectors.

Querying

With the store populated, the system can do semantic search. The query path is:

User Query
    ↓
Query Embedding
    ↓
ChromaDB Similarity Search
    ↓
Top-K Relevant Chunks
    ↓
Scores + Metadata
    ↓
Retrieved Context

Rather than relying wholly on exact keyword hits, semantic retrieval finds chunks that are conceptually related to the user's request. Results are then converted into the standardized retrieval contract the platform already uses.

Reconnecting the agent

Only when the retrieval infrastructure works does the full pipeline get wired back into RetrievalAgent. From the agent's point of view, the picture stays simple:

AgentRequest
     ↓
RetrievalAgent
     ↓
RetrievalService
     ↓
Relevant Documents
     ↓
Structured RetrievalResult

Behind RetrievalService, the implementation grows far more sophisticated:

RetrievalService
      ↓
Query Embedding
      ↓
ChromaDB
      ↓
Semantic Search
      ↓
Ranking
      ↓
Retrieved Context

The separation matters. The Retrieval Agent should treat retrieval as a capability and shouldn't need to know about vector databases, embedding generation, chunk storage or document ingestion.

The original agent worked on local data, and the architecture looked roughly like this:

RetrievalAgent
      ↓
RetrievalService
      ↓
Local Documents

The implementation can now evolve toward something richer:

RetrievalAgent
      ↓
RetrievalService
      ↓
Semantic Retrieval
      ↓
ChromaDB
      ↓
Embedded Enterprise Documents

Notice what stays fixed: the link from RetrievalAgent to RetrievalService. Only the infrastructure behind that contract changes, which is precisely why the retrieval abstraction was created before the vector database arrived. Infrastructure should be free to evolve without forcing a redesign of the agent architecture.

Fitting into the platform

The RAG pipeline doesn't exist on its own. Its purpose is to supply grounded enterprise knowledge to the larger multi-agent system.

User
                          │
                          ▼
                    Supervisor Agent
                          │
                          ▼
                     Planning Agent
                          │
                          ▼
                    Execution Plan
                          │
                          ▼
                     Agent Registry
                          │
             ┌────────────┼────────────┐
             ▼            ▼            ▼
       RetrievalAgent  ResearchAgent AnalysisAgent
             │
             ▼
      RetrievalService
             │
             ▼
         RAG Pipeline
             │
             ▼
      Relevant Context
             │
             ▼
       Downstream Agents

In that picture, the Retrieval Agent links the layer where multiple agents execute with the layer that holds enterprise knowledge.

One lesson deserves emphasis: RAG isn't simply "add a vector database." A sound retrieval architecture involves many decisions:

• Document contracts and ingestion
• Chunking and embedding generation
• Vector storage and metadata
• Semantic retrieval and ranking
• Source traceability
• Agent integration
• Testing and observability

The vector database is a single component inside a far bigger retrieval system.

A repetitive pattern, on purpose

The engineering loop for this milestone is intentionally repetitive:

Build One Layer
      ↓
Add Unit Tests
      ↓
Run Fast Suite
      ↓
Push to GitHub
      ↓
CI Validates
      ↓
Keep Green
      ↓
Build Next Layer

It lets complexity grow while confidence in the existing system holds. Rather than adding five components at once and debugging them together, each capability gets validated before the next arrives.

The first milestone of the real pipeline is complete: the SourceDocument contract, the DocumentChunk contract, tests for both models and a passing fast CI run. The roadmap that follows:

1. Document Contract             ✅ COMPLETE
          ↓
2. Chunking Service              ← NEXT
          ↓
3. Embedding Service
          ↓
4. ChromaDB Integration
          ↓
5. Semantic Retrieval
          ↓
6. RetrievalAgent Integration
          ↓
7. End-to-End RAG Pipeline

Each layer gets implemented and tested independently before moving on.

Why this matters

The first Retrieval Agent showed that the agent architecture worked. The goal now is to replace its temporary retrieval with real infrastructure, moving from local retrieval to a document pipeline, chunking, embeddings, vector storage, semantic search and finally a RAG-enabled agent. Since the agent depends on a service abstraction and not directly on ChromaDB, all of that can happen without redesigning the multi-agent platform.

A broader principle applies: build stable contracts around the components expected to change. Models will change, vector databases may be swapped, chunking strategies will evolve and retrieval algorithms will improve, but the platform around them shouldn't need rewriting each time.

What's being built is a retrieval capability that belongs inside an enterprise multi-agent platform, not a one-off RAG demo. There, the Supervisor orchestrates, the Registry resolves capabilities, specialized agents execute, RAG supplies grounded knowledge, tests protect behavior, CI guards every change and observability measures production. The method never varies: build a layer, test it, push it, keep CI green, then build the next one.