Semantic Retrieval with ChromaDB and Embeddings for a Multi-Agent Platform

A new milestone has landed in the Enterprise Multi-Agent AI Platform project: its retrieval pipeline now performs genuine semantic search, using embeddings stored in ChromaDB. The architecture has advanced from document preparation to a working retrieval path, running SourceDocument, then DocumentChunker, then EmbeddingService, then OpenAI Embeddings, then ChromaDB, then semantic search, and finally RetrievedDocument.

That is a meaningful change of stage. The platform no longer only prepares documents and generates embeddings. It can store vectorized enterprise knowledge, search it by meaning, rank the relevant chunks and hand back structured results. Building it also surfaced a few retrieval-quality problems that had to be fixed before the pipeline could be attached to the RetrievalAgent.

The pipeline takes shape

The retrieval architecture is laid out like this:

Enterprise Documents
        │
        ▼
   SourceDocument
        │
        ▼
 DocumentChunker
        │
        ▼
  DocumentChunk[]
        │
        ▼
 EmbeddingService
        │
        ▼
 OpenAI Embeddings
        │
        ▼
     ChromaDB
        │
        ▼
 Semantic Search
        │
        ▼
RetrievedDocument[]
        │
        ▼
  RetrievalAgent

Every layer owns one responsibility, and that is deliberate. The goal was never a single large RAG function that loads files, splits text, builds embeddings, manages the vector database, runs retrieval and produces an answer. The system is instead assembled from clear architectural boundaries.

Stage 1: source documents

All of it starts with enterprise knowledge. Over time that content may arrive from PDFs, DOCX and TXT files, financial reports, internal policies, technical documentation, research material and enterprise knowledge bases. The SourceDocument model gives that material one standard shape before it enters the pipeline. Conceptually it looks like this:

SourceDocument
 ├── document_id
 ├── content
 ├── source
 └── metadata

The principle that matters here is traceability. Whatever gets retrieved later must still link back to the source it came from.

Stage 2: chunking

Large documents generally shouldn't be embedded as one block. The DocumentChunker turns each SourceDocument into smaller pieces that are ready for retrieval:

SourceDocument
      │
      ▼
DocumentChunker
      │
      ├── Chunk 0
      ├── Chunk 1
      ├── Chunk 2
      └── Chunk N

Each chunk keeps its chunk_id, document_id, content, source, chunk_index and metadata. Semantic search can therefore work on small units of information without losing the tie to the original document.

Improving chunk quality

Running the real retrieval pipeline exposed a problem. Character-based chunking can cut ordinary prose at awkward points, sometimes straight through a word. A boundary might conceptually produce a chunk that ends like this:

...important financial risks for

followed by a chunk that begins like this:

the upcoming quarter...

or, in worse cases, a split in the middle of a word. Chunks like that are technically valid, yet retrieval quality is about more than valid data structures. Boundaries shape the semantic quality of whatever gets embedded.

The chunking strategy was therefore changed to prefer word boundaries for normal prose, while keeping a character-boundary fallback for text with no useful whitespace. The chunker now has two behaviors:

Normal prose
    ↓
Prefer word-aware boundaries

Text without whitespace
    ↓
Fallback to character boundaries

It's a small implementation detail with a big architectural lesson: passing tests doesn't mean retrieval quality is good, and real data reveals problems that isolated unit tests can't always show.

Stage 3: embeddings

Once chunks are ready, the EmbeddingService converts their text into numerical vectors.

DocumentChunk
      │
      ▼
EmbeddingService
      │
      ▼
Embedding Model
      │
      ▼
Vector Representation

The service supports single-text embeddings for queries and batch embeddings for document chunks. An earlier real local integration test confirmed a vector dimension of 1,536. The number itself matters less than the fact that the embedding implementation stays behind a service boundary, so the Retrieval Agent never needs to know how vectors are produced.

Stage 4: ChromaDB vector storage

The next layer is persistent vector storage. Each chunk can now be stored with its vector and metadata. Conceptually:

DocumentChunk
      +
Embedding
      +
Metadata
      │
      ▼
   ChromaDB
      │
      ▼
Persistent Vector Collection

A vector record can keep information such as:

Vector Record
 ├── chunk_id
 ├── embedding
 ├── content
 ├── document_id
 ├── source
 ├── chunk_index
 └── metadata

A bare vector is no use to the rest of the application. The platform needs the actual content back and needs to know where that content came from.

Stage 5: semantic search

With embedded chunks sitting in ChromaDB, the system can do semantic retrieval. The query path looks like this:

User Query
     │
     ▼
EmbeddingService
     │
     ▼
Query Embedding
     │
     ▼
   ChromaDB
     │
     ▼
Similarity Search
     │
     ▼
Top-K Relevant Chunks
     │
     ▼
RetrievedDocument[]

Retrieval changes from finding documents that contain exactly these words to finding chunks whose meaning best fits the query. That is one of the core capabilities behind RAG.

A correction to the similarity metric

Real testing exposed a second issue. Vector databases support different distance or similarity metrics, and if the collection uses one metric while the application reads the returned distance as another, the relevance score becomes misleading. For this pipeline the ChromaDB collection was explicitly configured for cosine distance, so the retrieval layer can consistently read results as cosine-based similarity.

Query Embedding
      │
      ▼
ChromaDB
      │
      ▼
Cosine Distance
      │
      ▼
Similarity Score
      │
      ▼
Ranked Results

Semantic retrieval isn't only about returning documents. It's also about understanding how relevant they are.

Real retrieval results

After the chunking and similarity fixes, the pipeline was tested with a real financial-risk query. The ranked results came back with similarity scores of roughly 0.6181, 0.5520 and 0.4198, and the retrieved context included the relevant risk information on liquidity risk and market volatility. The pipeline therefore made it all the way through this path:

Question
   ↓
Query Embedding
   ↓
ChromaDB
   ↓
Semantic Similarity
   ↓
Relevant Chunks
   ↓
Financial Risk Context

At that moment the retrieval infrastructure stopped being setup work and began to show real semantic knowledge retrieval.

Why real retrieval testing matters

Mocked unit tests stay extremely valuable. They give speed, determinism, CI safety, zero external API cost, controlled failure scenarios and isolation of application logic. They can't answer every question, though. A mocked test can confirm that this works structurally:

EmbeddingService
      ↓
Expected Vector

but it can't fully show that the following chain yields good retrieval behavior:

Real Document
      ↓
Real Chunking
      ↓
Real Embedding
      ↓
Real Vector Database
      ↓
Real Semantic Query
      ↓
Useful Context

So the testing strategy keeps several levels of validation apart.

Testing strategy

The platform is moving toward three complementary layers of testing.

Unit tests are fast and deterministic and belong in the normal CI pipeline:

Application Logic
      ↓
Mock External Dependencies
      ↓
Controlled Results
      ↓
Assertions

Integration tests validate real infrastructure and can run separately from the fast CI suite:

Application
      ↓
Real Embedding Provider
      ↓
Real ChromaDB
      ↓
End-to-End Retrieval

Performance and resilience testing becomes necessary as the platform matures:

Realistic Workload
      ↓
Concurrent Requests
      ↓
Latency
Throughput
Rate Limits
Failures
Cost
Scalability

A production-ready AI system needs all three. Mocking external services in unit tests doesn't replace integration testing, and integration testing doesn't replace performance testing. Each layer answers a different engineering question.

CI stays green

After the retrieval improvements the fast suite is healthy, with 17 tests passing and 2 integration tests deselected. That matters because the RAG pipeline is getting more sophisticated without giving up deterministic CI validation. The development loop continues as follows:

Code Change
     ↓
Fast Unit Tests
     ↓
Commit
     ↓
Push
     ↓
GitHub Actions
     ↓
CI Validation
     ↓
Keep Green

Tests against real services stay available as a separate run.

Connecting RAG to the multi-agent platform

The retrieval infrastructure doesn't exist in isolation. It is meant to become the knowledge layer for the larger multi-agent system, and the broader architecture is heading this way:

User
                          │
                          ▼
                    Supervisor Agent
                          │
                          ▼
                     Planning Agent
                          │
                          ▼
                    Execution Plan
                          │
                          ▼
                     Agent Registry
                          │
             ┌────────────┼────────────┐
             ▼            ▼            ▼
      RetrievalAgent ResearchAgent AnalysisAgent
             │
             ▼
      RetrievalService
             │
             ▼
      EmbeddingService
             │
             ▼
          ChromaDB
             │
             ▼
      Semantic Search
             │
             ▼
      Relevant Context
             │
             ▼
       Downstream Agents

This is the point where RAG and multi-agent orchestration start to merge.

Keeping the Retrieval Agent simple

A key goal is to keep the RetrievalAgent focused on its own job. It shouldn't need to know about chunking algorithms, the embedding implementation, vector dimensions, ChromaDB internals, similarity calculations or storage configuration. Instead:

RetrievalAgent
      ↓
RetrievalService
      ↓
VectorStoreService
      ↓
EmbeddingService + ChromaDB

The agent asks for relevant knowledge, and the retrieval infrastructure decides how that knowledge is found. This separation lets the underlying implementation evolve without a rewrite of the agent.

Next milestone: replace the temporary retrieval

The next step is clear. The Retrieval Agent still has to be connected directly to the real retrieval infrastructure. Today it looks like this:

BEFORE

RetrievalAgent
      ↓
Temporary Local Documents

and the target is this:

NEXT

RetrievalAgent
      ↓
RetrievalService
      ↓
VectorStoreService
      ↓
EmbeddingService
      ↓
ChromaDB
      ↓
Semantic Search
      ↓
Enterprise Knowledge

After that integration the agent stops returning temporary hard-coded results and pulls knowledge from the real indexed collection of documents.

From infrastructure to multi-agent execution

That integration marks another transition. Most of the work so far has gone into building the retrieval infrastructure. Once RetrievalAgent consumes the real semantic pipeline, the architecture can support workflows like this:

User Request
      ↓
Supervisor
      ↓
Planning Agent
      ↓
Execution Plan
      ↓
Retrieval Agent
      ↓
Enterprise Knowledge
      ↓
Analysis Agent
      ↓
Report Agent
      ↓
Quality Review
      ↓
Final Response

RAG then becomes a capability inside a broader multi-agent execution system and no longer a standalone retrieval feature.

The road ahead

The retrieval roadmap moves from foundational components toward production-oriented capabilities:

Document Contracts               ✅
        ↓
Chunking                         ✅
        ↓
Embeddings                       ✅
        ↓
ChromaDB Vector Storage          ✅
        ↓
Semantic Search                  ✅
        ↓
RetrievalAgent Integration       ← NEXT
        ↓
Reranking
        ↓
Caching & Cost Controls
        ↓
Observability & Metrics
        ↓
Multi-Tenant Isolation
        ↓
Production Hardening

The later stages grow in importance as the platform shifts from functional architecture to enterprise operation.

Production RAG needs more than retrieval

Getting relevant chunks out of ChromaDB is a milestone and not the end of the architecture. A production retrieval platform eventually has to handle:

• Reranking, because initial vector similarity doesn't always give the best ordering.
• Caching, since repeated embeddings and searches add latency and cost.
• Observability, so retrieval latency, embedding latency, document counts, scores, failures and cost can all be measured.
• Model and vector-space evolution, because switching embedding models may call for versioned indexes and controlled re-embedding.
• Rate-limit handling, since external AI providers impose operational constraints that aren't deterministic.
• Multi-tenant isolation, because enterprise data can't share unrestricted retrieval boundaries.
• Schema contracts, meaning stable interfaces between ingestion, embedding, storage, retrieval and agent layers.
• Performance testing under realistic concurrency and data volume.

These belong on the roadmap, but they should be added around stable boundaries and not mixed into one large implementation from day one.

The engineering principle

The biggest lesson of this stage: design the abstraction first, check behavior against real data, and then harden the production traits as the architecture grows. The progression has been deliberate:

Data Contract
     ↓
Chunking
     ↓
Unit Testing
     ↓
Embeddings
     ↓
Mocked Provider Testing
     ↓
Real Integration Validation
     ↓
Vector Storage
     ↓
Semantic Retrieval
     ↓
Retrieval Quality Improvements
     ↓
Agent Integration

Each stage exposes a different class of problem, which is exactly why incremental architecture pays off.

From documents to multi-agent execution

The larger vision is coming into focus:

Enterprise Knowledge
        ↓
Document Processing
        ↓
Chunking
        ↓
Embeddings
        ↓
Vector Storage
        ↓
Semantic Retrieval
        ↓
Retrieval Agent
        ↓
Multi-Agent Orchestration
        ↓
Analysis + Research + Reporting
        ↓
Quality Review
        ↓
Grounded Enterprise Response

RAG isn't the entire platform. It is the knowledge infrastructure that lets specialized agents work on grounded enterprise context.

Current milestone

What exists today: uniform document models, plain and word-boundary-aware chunking, an embedding service with mocked unit tests plus a real validation run, persistent ChromaDB storage with cosine similarity set explicitly, semantic search that returns ranked results, and a fast suite of 17 passing tests. The next milestone is wiring RetrievalAgent to the real semantic retrieval pipeline, which is where the system stops merely containing a RAG subsystem and starts running RAG-powered multi-agent execution.

Enterprise AI doesn't mean plugging an LLM into a vector database and declaring victory. Layers need contracts, contracts need tests, integrations need real validation, and the system needs room to evolve. The pipeline has grown from documents, chunks, embeddings and vectors into documents, chunks, embeddings, ChromaDB, semantic search and relevant enterprise context. What comes next is that context flowing into the RetrievalAgent and then into multi-agent execution. The working loop stays constant: build, test, measure, improve, integrate and scale.