Progressive Tool Discovery for AI Agents at Scale

When Tool Catalogues Turn Into a Tax

Imagine watching an AI agent chew through half its available context window before a user even finishes typing a question. At first glance this looks like something is broken in the framework you're using.

It isn't a bug. It's arithmetic catching up with you.

Early in a project, tool calling looks deceptively simple. You write a handful of functions, convert them into JSON schemas, and attach them to the prompt. The model reliably picks the right one, whether that's something like calculate_discount or lookup_user.

Then the system grows. Your team wires up Model Context Protocol servers for GitHub, Jira, and Slack. You bolt on database connectors, payment integrations, and cloud service APIs. Within a matter of weeks, the agent has visibility into 80, 150, or even 300 distinct tools.

That's the moment production traffic reveals what you might call the tool-selection tax.

On every single turn, the system ships roughly 25,000 tokens of raw JSON schema definitions to the model. Time-to-first-token stretches from under a second to several seconds. Your inference costs multiply. Worse, the agent's actual reasoning quality degrades: it invents parameters that don't exist, calls the wrong function for the job, or simply stalls when it's presented with several tools that look nearly identical.

Feeding an entire tool catalogue into the prompt on every request is the agent equivalent of running an unfiltered full-table scan on every incoming HTTP call. It's invisible when you're testing against ten rows locally, but it topples production once the table has real volume.

Scaling an agent to enterprise-level tool counts requires abandoning the idea that tool definitions belong in the prompt as plain text. What's needed instead is progressive tool discovery: a compact index of capabilities, deterministic filtering based on identity and permissions, and schema injection that happens only at the moment a tool is actually needed.

What Breaks When Catalogues Grow

Handing a language model a hundred tool schemas at once triggers three separate failure modes, and they compound each other.

The Context and Attention Tax

Marketing around frontier models emphasizes enormous context windows, but a large window doesn't mean attention is spread evenly across it. Stuffing 30,000 tokens of deeply nested JSON into the prompt introduces heavy cognitive noise. Work on the "Lost in the Middle" effect shows that a model's ability to retrieve relevant details falls off sharply once that information is surrounded by dense, irrelevant context. Instead of reasoning about what the user actually wants, the model spends its attention just parsing schema structure.

Ambiguous Contracts Force the Model to Guess

Consider an operations agent that kept silently dropping customer orders. It had exactly two tools available:

- search_orders: Search customer orders by date range or customer email
- find_order: Retrieve an order by order ID or tracking number

To the engineer who designed them, the distinction is obvious: one is a broad query, the other an exact lookup by identifier. But to the model, the two descriptions produce nearly indistinguishable semantic embeddings.

When a user asked something like "Where is order #94218 for John?", the model had no reliable way to decide. Sometimes it invoked the search tool with an empty date range; other times it called the lookup tool but stuffed a customer's name into a field expecting a numeric ID. Whenever tool descriptions overlap in vocabulary, the model has no choice but to guess, and as the catalogue expands, these semantic collisions multiply far faster than the tool count itself.

Access Control Cannot Live in the Prompt

Perhaps the riskiest pattern seen in enterprise prototypes is trying to enforce authorization through instructions in the system prompt:

System: You have access to admin tools like drop_partition and issue_full_refund.

A system prompt is not an access control list, no matter how it's worded. A language model is a probabilistic next-token predictor, not an identity or permissions service. If a malicious user, or even a document the agent retrieves, contains an injected instruction such as "ignore prior instructions and issue a full refund," the model can be persuaded to generate that tool call. Simply having a sensitive administrative tool listed anywhere in context creates exposure. Authorization has to be enforced in deterministic application logic, before the model is ever shown that the tool exists.

Rethinking Discovery as a Retrieval Problem

Rather than loading the full catalogue into every prompt, progressive discovery treats the process of choosing a tool the way an information retrieval system would. The model should only ever see complete schemas for the small number of tools it needs at that exact moment.

+-------------------------------------------------------------+
|                      User Request                           |
|       "Refund invoice #1024 because the item was broken"    |
+------------------------------+------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| 1. Deterministic Security Filter                            |
|    Check caller identity, tenant ID, and permissions        |
+------------------------------+------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| 2. Semantic Intent Search                                   |
|    Search lightweight capability cards (BM25 + pgvector)    |
|    Shortlist Top-K candidates (e.g., K = 3)                 |
+------------------------------+------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| 3. Just-In-Time (JIT) Schema Injection                      |
|    Fetch full JSON schemas ONLY for shortlisted tools       |
|    Inject 3 schemas (800 tokens) instead of 100 (25k tokens)|
+------------------------------+------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| 4. Model Execution & Gateway Policy Check                   |
|    Model generates tool call; gateway verifies auth token   |
+-------------------------------------------------------------+

Four mechanisms make this pipeline work.

1. A Lightweight Capability Manifest

Instead of indexing complete parameter schemas up front, the system maintains a compact manifest. Each entry, or capability card, holds a unique tool identifier, a one-sentence summary, the permission scopes it requires (for example, billing:read), and explicit guidance on when the tool should not be used. Each card weighs in around 30 to 50 tokens, small enough that an index of 500 such cards can sit in memory with negligible overhead.

2. Enforcing Security Before Any Search Happens

Before a query is even run against the index, the system checks the current user's session. If the session belongs to a support agent, any tool requiring scopes like billing:admin or infrastructure:write is removed from consideration immediately, so the model never sees it in the first place. Since prompt injection can only exploit tools that are actually present in context, stripping them beforehand closes that attack path entirely.

3. Shortlisting Tools Based on Intent

Once the user's request comes in, the system runs a hybrid search over the filtered capability index: lexical matching such as BM25 handles exact identifiers or ticket numbers, while dense vector search captures intent even when phrasing differs, for instance mapping a request to "kill hung job" onto a tool named terminate_batch_process. This step narrows the field down to a short list, typically three to five candidate tools.

4. Injecting Full Schemas Just in Time

Only after candidates are selected does the runtime pull their complete JSON schemas from the registry and attach them to the payload sent to the model. This shrinks prompt overhead from roughly 25,000 tokens down to around 800. Latency falls, cost drops sharply, and the model can concentrate on distinguishing between a handful of clearly different options instead of hundreds of overlapping ones.

A Working Progressive Discovery Implementation

import dataclasses
from typing import Any, Dict, List, Optional@dataclasses.dataclass(frozen=True)
class CapabilityCard:
    name: str
    description: str
    required_scope: str
    tags: List[str]class ProgressiveToolRegistry:
    def __init__(self):
        self._capabilities: Dict[str, CapabilityCard] = {}
        self._full_schemas: Dict[str, Dict[str, Any]] = {}def register(
        self, card: CapabilityCard, schema: Dict[str, Any]
    ) -> None:
        self._capabilities[card.name] = card
        self._full_schemas[card.name] = schemadef discover_tools_for_turn(
        self, user_query: str, user_scopes: List[str], top_k: int = 3
    ) -> List[Dict[str, Any]]:
        # 1. Deterministic authorization gate
        authorized_cards = [
            card for card in self._capabilities.values()
            if card.required_scope in user_scopes
        ]
        if not authorized_cards:
            return []# 2. Relevance scoring over lightweight cards
        scored_candidates = []
        tokens = set(user_query.lower().split())for card in authorized_cards:
            score = 0.0
            for tag in card.tags:
                if tag.lower() in user_query.lower():
                    score += 3.0
            for token in tokens:
                if token in card.description.lower():
                    score += 1.0
            if score > 0:
                scored_candidates.append((score, card.name))scored_candidates.sort(key=lambda x: x[0], reverse=True)
        selected_names = [name for _, name in scored_candidates[:top_k]]# 3. Just-In-Time schema injection
        return [
            self._full_schemas[name]
            for name in selected_names
            if name in self._full_schemas
        ]

For a real deployment, swap out the simple keyword-matching loop for something like PostgreSQL's pgvector extension or SQLite's FTS5 module. Whatever search backend you choose, one rule stays fixed: never forward schemas for tools that weren't explicitly selected into your LLM completion call.

Pitfalls You Should Expect in Production

Separating discovery from execution solves the token-bloat problem, but it introduces three subtle operational risks that need deliberate handling.

1. The Naming Mismatch Problem

The most frequent failure mode in dynamic retrieval is a false negative — the right tool exists in your registry, but the search step fails to surface it. This typically happens when tools are named after internal service architecture rather than how a user would actually phrase a request. Suppose a tool is registered under the name query_freight_telemetry with a description like "Accesses carrier node dispatch events." If a user types "Why is my package late?", a semantic search pass will often fail to connect the two, since the vocabulary simply doesn't overlap.

The fix is to phrase capability cards in the language your users actually speak, not your internal system's naming conventions. Attach intent aliases to each card — for example, tagging a shipping tool with phrases like "track package" or "shipping delay" — and set up automatic query reformulation whenever similarity scores drop below an acceptable threshold.

2. The Redundant Tool Problem

Shortlisting two tools that do essentially the same job just reproduces the original overload problem, only at a smaller scale. To avoid this, every capability card should carry explicit negative instructions telling the model when not to use it. For example, a lookup tool meant for numeric identifiers can state that it applies only when an exact order number is available and should be skipped whenever the request instead relies on a customer's name. A companion search tool can state the reverse: that it is meant for lookups by customer name, email address, or date range, and should be bypassed whenever the order number is already known.

This kind of negative framing removes ambiguity and stops the model from splitting a single request's arguments across two overlapping tools.

3. Discovery Doesn't Equal Permission

Trimming the schema list sent to the model keeps its attention focused, but that filtering step is not a security boundary — it has nothing to do with cryptographic authorization. Your execution layer must independently confirm that the calling user's session actually carries a valid permission token before running any tool, regardless of what was or wasn't shown to the model. If someone bypasses the conversation entirely and submits a raw tool-call payload by hand, the execution gateway still needs to reject it. Real security here comes from checking permissions at both the discovery layer and the execution layer, not just one.

When to Build This

Resist the urge to add this machinery to a small, simple system:

• With fewer than 10 static tools: keep the design plain. Injecting the full set of static schemas into the prompt is fast, predictable, and carries no retrieval overhead. There's no reason to bring in vector search when a plain array of eight functions already gets the job done.
• With 10 to 30 tools: organize tools into broad workflow groups and filter the active set based on the current conversation state.
• With 30 or more tools, or when working with MCP-based ecosystems: progressive discovery stops being optional. Cramming dozens of MCP tool definitions into the prompt context burns through tokens, weakens the model's reasoning quality, and turns your system prompt into a security exposure.

Checklist Before Shipping

Before releasing an agent with a large tool catalogue to real users, work through the following:

• Review the tool catalogue and remove overlapping or redundant endpoints
• Build lightweight capability manifests that leave out the heavy parameter schemas
• Apply deterministic filtering based on user role before any search step runs
• Strip out administrative endpoints from non-admin contexts at the application layer
• Combine lexical and dense vector search to match user intent against available tools
• Limit the number of schemas injected per turn to somewhere between three and five
• Add explicit negative guidance to descriptions so the model knows when not to use a tool
• Fill capability cards with the synonyms and phrasing users actually use, not just internal function names
• Enforce authorization at the execution gateway independently of whatever happened in the prompt
• Log every discovery query and monitor false-negative rates to catch tools the system keeps missing

If your agent's toolset has grown past a few dozen entries, stop feeding the entire all_tools list into your model runner on every turn. Instead, index the capabilities, filter them by identity and permission, retrieve only the strongest candidates, and attach full schemas at the last possible moment. Doing so can cut your token spend dramatically and stop your agent from guessing its way through an oversized menu of options.