Chroma Context-1: A 20B Search Sub-Agent for Cheaper, Sharper Agentic RAG

In most RAG systems, retrieval is the weak link. You can put GPT-4, Claude, Gemini or any frontier model on top, and if the search step returns junk, the model will still produce confident, polished, wrong answers.

Consider a legal document search where a question's answer is spread across three filings, and the agent has to locate all three. The usual build is a frontier model in a ReAct loop with vector search tools. It works, but at something like $0.12 and fifteen seconds per query, it's a demo and not a product feature. Chroma's Context-1 targets exactly that cost and latency problem.

What Context-1 is

Context-1 is a 20-billion-parameter language model made for one job: retrieval. It doesn't generate final answers or do general reasoning. It searches.

In a typical agentic RAG setup, one frontier model does everything: it plans what to search, runs the search, decides whether it needs more, and writes the answer. That's like hiring a brain surgeon to do the grocery shopping too. Context-1 instead acts as a dedicated search sub-agent that finds the documents, and a separate frontier model handles reasoning and writing. It is separation of concerns applied to AI architecture.

The model is built on gpt-oss-20B, an open base model, and trained with supervised fine-tuning followed by reinforcement learning using Chroma's CISPO method (Clipped Importance-Sampled Policy Optimization). The outcome is a small model that performs well above its size on retrieval benchmarks.

Why ordinary agentic RAG falls short

A classic pipeline makes a single hop: the question is embedded, the top-K chunks come back and an LLM writes the answer.

User Query → Embedding Search → Top-K Chunks → LLM → Response

That has well-known problems. You receive chunks and never full documents, so global context is lost. A single pass can't link facts spread over several documents. And semantic similarity isn't the same as relevance, since a chunk can share vocabulary with the question and still not answer it.

The agentic upgrade wraps this in a ReAct loop, where the model plans, searches, looks at the results and decides whether to search again.

User Query → Plan → Search → Observe → Need more info? → Search again → Generate

Better, but there's a catch. The same large frontier model carries out every step: planning, calling tools, judging results and pruning irrelevant chunks, all at frontier prices. Smaller models can't stand in, because the system prompt and tool definitions alone confuse them and they don't follow the loop reliably. You end up paying premium rates for work that doesn't need premium intelligence.

How it works

Chroma didn't just fine-tune a model. It built an agent harness and trained the model inside it, so the model and its tools come as a pair. Four tools are part of the training:

• search_corpus runs hybrid search, combining BM25 keyword matching with dense vector search through reciprocal rank fusion, so you get exact matches and semantic understanding together.
• grep_corpus does regex matching across the corpus and returns up to five matching chunks, handy for a specific date, name or case number.
• read_document fetches a full document by ID when a chunk looks promising.
• prune_chunks lets the model delete chunks from its own context when it decides they're irrelevant. This is the standout.

A trace shows the pattern. On turn one the model searches for something like merger and acquisition penalties, and eight chunks come back while the token counter reads 8,432 of 32,768. On turn two it reads a document in full and searches again, then prunes whatever didn't help, freeing room to look further.

Turn 1: search_corpus("merger acquisition penalties 2024")
  → Retrieves 8 chunks [Token usage: 8,432/32,768]
Turn 2: read_document("doc_14") + search_corpus("SEC filing penalties")
  → Retrieves 4 more chunks [Token usage: 18,203/32,768]
Turn 3: prune_chunks(["chunk_3", "chunk_7", "chunk_9", "chunk_11"])
  → Removes 4 irrelevant chunks [Token usage: 14,203/32,768]
Turn 4: search_corpus("specific penalty amounts regulatory action")
  → Retrieves 3 final chunks [Token usage: 21,847/32,768]
→ Returns ranked supporting documents to the reasoning model

The self-editing context

The habit in most pipelines is to pull back a batch of chunks, drop them all into the prompt and trust the model to look past the ones that don't matter. Research shows models don't just ignore noise, they get confused by it, an effect called context rot. Context-1 addresses it directly. Its context window is 32,000 tokens and the harness monitors usage as it goes:

1. After every tool call the model sees its current usage, in a form like [Token usage: 14,203/32,768].
2. Around 24K tokens, the harness gives a nudge to prune or wrap up.
3. Past a configurable hard limit, the model may only prune or finish, with no more retrieval, which prevents runaway context growth.
4. The pruning decision belongs to the model itself. No separate reranker or heuristic is involved, and the one that fetched the chunks judges them.

Prune accuracy reaches 0.941, up from 0.824 in the base model, so it spots irrelevant chunks about 94% of the time. This goes further than Naver Labs' Provence, which used a dedicated model to prune sentences. Here the search agent prunes as part of its normal workflow, with no extra model.

Three tiers

Chroma stresses, and it's sound advice, that Context-1 should be used for retrieval only, not for generation. The architecture has three tiers: your data layer, the Context-1 search sub-agent, and a frontier reasoning model on top. The split pays off in four ways:

• Each part can be improved on its own, so you can swap in a better retriever tomorrow without touching generation.
• The expensive model only sees clean, curated context, not raw search dumps.
• Context-1 generates 400 to 500 tokens per second on an Nvidia B200, which is quick for multi-hop search.
• The reasoner gets focused material and produces better answers.

It's the microservices principle again: don't make one model do it all.

How it was trained

Supervised fine-tuning. The teacher was Kimi K2.5, a large frontier model. The team generated search trajectories with it inside the four-tool harness, filtered them by recall and trained Context-1 to imitate both successful and unsuccessful runs. The failed runs were kept on purpose, because even low-recall searches contain well-formed tool calls, query decomposition and pruning choices. The student learns the mechanics of searching from searches that missed things.

Reinforcement learning with CISPO. Each training step used 128 queries with 8 rollouts each, so 1,024 trajectories, across 5 epochs, converging near step 230. Every token contributes to learning, including rare but critical ones such as pruning decisions. The reward blends several terms:

Reward = F1_score (recall-weighted)
       + trajectory_recall_bonus
       + final_answer_bonus (+1.0)
       - repeated_pruning_penalty (0.1 per excess)
       - turn_count_penalty

Training also followed a two-stage curriculum. Early on it used easier single-hop tasks with recall weighted 16 times over precision, which teaches aggressive exploration. Later it moved to harder multi-hop tasks with a 4x weighting, which teaches selectivity. It resembles training a search dog: first to find any scent, then the specific one.

The data pipeline

One of the most useful things Chroma released may be its synthetic data generation pipeline, open-sourced on GitHub, which works as a blueprint for building retrieval training data in any domain. It follows an explore, verify, extend pattern:

Seed Topic → Explore Web → Extract Verifiable Facts → Generate Tasks
     ↓
Add Distractors → Chain Into Multi-Hop → Verify Answers

1. Exploration. From a seed topic, such as a random Wikipedia title, an agent browses the web and collects documents with unique, checkable facts.
2. Task generation. Documents become clues and questions that need real multi-step reasoning, not simple lookup.
3. Distractors. The pipeline writes passages that resemble the right answer but point elsewhere, imitating real retrieval noise.
4. Chaining. Single-hop tasks are linked by using one answer to ask the next question, producing multi-hop challenges one to four hops deep.

It supports four domains out of the box: the web, SEC filings for finance, USPTO patents for legal work, and email, using Enron-style datasets. Trust in the data comes from extraction-based verification, where LLM judges pull quotes from documents and matching clue quotes, normalize them and confirm the text is grounded. Alignment accuracy exceeds 80% in every domain, and human checking shrinks from reading whole documents to comparing quote pairs.

Trying it today

There are three routes.

The hosted API is the easiest because the harness is handled for you, though it sits behind a waitlist.

The weights are on Hugging Face under the Apache 2.0 license:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "chromadb/context-1",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("chromadb/context-1")

The big caveat is that the harness itself, meaning tool execution, token budgets, context pruning and deduplication, isn't public yet. Without it you won't reproduce the technical report's results. The pattern is familiar from coding agents: the same model behaves very differently inside different tools, because the harness makes the difference. Context-1 without it is a Formula 1 engine without the car. Chroma says it plans to publish the harness and the evaluation code.

The third route is writing your own harness around the four documented tools. This pseudocode gives the idea and won't match Chroma's production version:

# Pseudocode for a basic Context-1 harness

tools = {
    "search_corpus": hybrid_bm25_vector_search,    # BM25 + dense vector via RRF
    "grep_corpus": regex_pattern_search,             # Regex matching, max 5 chunks
    "read_document": full_document_retrieval,        # Full doc by ID
    "prune_chunks": context_pruner                   # Remove chunks from context
}
context_budget = 32_768  # tokens
soft_threshold = 24_000
current_usage = 0
while not done:
    # Get model's next action
    response = model.generate(conversation_history)
    # Execute tool calls (model may call multiple in parallel)
    for tool_call in response.tool_calls:
        result = tools[tool_call.name](**tool_call.args)
        conversation_history.append(result)
        current_usage = count_tokens(conversation_history)
    # Enforce context budget
    if current_usage > soft_threshold:
        # Suggest pruning or concluding
        conversation_history.append(
            f"[Token usage: {current_usage}/{context_budget}] "
            "Consider pruning irrelevant chunks or concluding search."
        )

Current limits

• It's trained for depth, finding the needle. Breadth queries that must find every document matching a criterion aren't its strength.
• It writes no code, so it can't produce SQL to query structured data. It searches documents only.
• It doesn't learn your data's structure or metadata, so schema discovery is up to your infrastructure.
• Without the official harness, the reported results aren't achievable, which blocks many teams for now.
• A 32K window, even with self-editing, may be too little working memory for some tasks.

What it suggests

Context-1 matters beyond Chroma. Through the last two years the habit has been to pick the biggest model your budget allows and let it handle every step. Context-1 suggests that small specialized models can match frontier ones on narrow tasks, and that combining specialists, such as a retriever, a reasoner and a code generator, gives you a system that runs faster and costs less, and frequently answers better than one monolithic model. Software went through the same shift when large monolithic apps were split into microservices. Monoliths still ran fine. Small focused components just scaled more easily.

For RAG builders in 2026 the useful question has changed. Skip asking which frontier model should run everything, and ask which specialist suits each stage. A 20B model keeping pace with GPT-5.4 and Opus-4.5 at retrieval casts doubt on the notion that bigger is always better that bigger is always better, which the marketing has pushed.