Reference Architecture for Production-Grade Enterprise Agentic AI Systems

Executive Summary

Moving from stateless Large Language Model (LLM) applications toward production-grade Agentic AI marks a major shift in how enterprises build software. Early enterprise AI efforts leaned heavily on basic Retrieval-Augmented Generation (RAG) — pulling documents into embeddings and stuffing them into a prompt's context window. That approach works fine for straightforward question-answering, but it falls short on autonomous decision-making, multi-step stateful planning, resilient tool invocation, and the ability to self-correct across a workflow.

Enterprise Agentic AI closes that gap by repositioning foundation models: instead of acting as the direct interface to a user, the model becomes a reasoning component embedded inside a deterministic software harness. Such systems can sense their environment, break large goals into smaller steps, call internal enterprise services, and carry execution state across multi-step transactions. Getting from a working prototype to something you can run in production, though, means solving for flaky non-deterministic control loops, degradation of context over long sessions, privilege-escalation risks, and runaway token spend.

This piece lays out an end-to-end reference architecture aimed at architects and AI engineering leads responsible for enterprise agent systems. It covers the foundational layers any agent needs, compares multi-agent topologies, dives into integration patterns for enterprise-grade retrieval — with particular attention to Amazon Kendra — and includes working Python examples suited for production use.

Section 1: The Canonical Enterprise Agent Architecture

Core Layers: Perception, Reasoning, Planning, Memory, and Tool Execution

A production-ready agent system keeps the probabilistic foundation model separate from the deterministic execution runtime that surrounds it. Five core layers typically make up this architecture, all running inside a managed runtime container:

• Perception Layer: Converts raw enterprise signals — telemetry, user messages, webhook payloads, API responses — into clean, typed inputs the rest of the system can consume. This includes input sanitization, rate-limiting, and early schema checks, all performed before the LLM is even invoked.
• Reasoning Engine: The cognitive center, powered by an underlying LLM (for example, Anthropic Claude 3.5 Sonnet or AWS Bedrock models). Rather than taking action itself, this engine interprets the current context, weighs possible paths forward, and emits structured statements of intent.
• Planning Module: Handles goal decomposition, turning a high-level instruction from a user into a structured, executable Directed Acyclic Graph (DAG) of steps. It can dynamically re-plan when a tool call fails or returns something unexpected or incomplete.
• State & Memory Management: Keeps track of three memory tiers — transient working memory (the live scratchpad for the current thread), a short-term log of the execution trajectory, and longer-lived semantic or episodic memory that spans multiple user sessions.
• Tool Execution & Governance Layer: Converts the agent's structured intents into real-world effects — outbound API calls, SQL queries, or remote function invocations. This is also where static sandboxing, parameter validation, role-based access control (RBAC), and circuit-breaking logic live.

Orchestration Patterns: Single-Agent Loops vs. Multi-Agent Topologies

Choosing the right orchestration pattern depends heavily on how complex the target domain is:

• Single-Agent ReAct Loops: One reasoning loop cycling through Thought, Action, and Observation phases. This suits narrow, linear workflows that touch fewer than 5 to 8 distinct tools. Beyond that, a single agent tends to run into bloated context windows, drifting instructions, and confusion over which tool to pick.
• Supervisor / Leader Multi-Agent Topology: A layered arrangement in which a top-level "Supervisor" agent takes in the original request, splits it into sub-tasks, and hands those off to specialized agents (say, a SQL agent, a RAG search agent, and an action-execution agent). The supervisor owns the global state, while each specialized agent works with a narrow, purpose-built set of tools.
• Peer-to-Peer / Network Topology: Specialized agents talk directly to one another over a shared event bus, with no central coordinator. This offers a lot of flexibility but tends to introduce non-determinism, the risk of message loops that never terminate, and debugging headaches that are hard to justify in an enterprise context.

For enterprise production use, the Supervisor Multi-Agent Topology is the recommended default, thanks to its clean state boundaries, easier auditability, and more predictable context costs.

Section 2: Enterprise Memory Management & Context Hygiene

Memory Hierarchy: Working Memory, Short-Term Trajectory, and Long-Term Memory

Think of agent memory management as a resource-budgeting problem. Letting memory grow unchecked drives up inference cost, adds latency, and causes the model to lose the thread as context degrades. A well-designed enterprise agent needs a layered memory structure:

• Working Memory (Scratchpad): The live context window — system instructions currently in effect, the state of the active task, and the most recent tool outputs.
• Short-Term Trajectory Memory: A temporary store, such as Redis or DynamoDB, holding the raw event history for the current session — full tool call payloads and their raw responses.
• Long-Term Memory: Durable storage — relational, vector, or graph databases — that retains summarized past interactions, user preference profiles, and domain-specific lessons learned across many sessions.

Strategies for Context Compaction, Pruning, and State Isolation

Keeping context from rotting requires the runtime to actively enforce hygiene rules:

• Observation Truncation: Raw tool responses — like a JSON payload with 500 records — should never go straight into working memory. The runtime needs to clean, truncate, or summarize what a tool returns before it reaches the reasoning engine.
• Rolling Window Compaction: Once the scratchpad crosses a set budget threshold (for instance, 20% of the model's total context limit), a compaction step condenses earlier turns of the conversation into compact semantic summaries and drops the original raw exchanges.
• Sub-Task State Isolation: When the supervisor hands a task to a sub-agent, it builds that sub-agent a fresh context containing only the specific goal and parameters it needs — none of the supervisor's own reasoning history leaks through.

Section 3: Deep Dive: Enterprise Retrieval Grounding via Amazon Kendra

Amazon Kendra as a Production Retrieval Layer

Building a retrieval pipeline on a generic vector database usually means engineering your own document ingestion, chunking logic, embedding generation, and a hybrid search fusion layer from scratch. Amazon Kendra instead offers a fully managed enterprise search engine that handles this out of the box. It comes with built-in multi-stage natural language understanding, structural parsing that respects tables, headers, and footers, and a hybrid ranking model that blends lexical and semantic signals.

Inside an enterprise agent stack, Kendra typically acts as the core Knowledge Grounding Subsystem — the component responsible for letting agents pull verified factual context from disparate internal data sources, without the chunk-fragmentation risk you get from naive vector-store splitting.

Advanced Relevance Tuning and Dynamic Metadata Trimming

To support autonomous decision-making safely, enterprise-grade retrieval needs precise permission handling and tunable relevance controls:

• Native access control lists (ACLs): Kendra automatically pulls in and maps the document-level ACLs defined in the originating systems, whether that's SharePoint, Confluence, or S3. When an agent issues a query, it forwards the authenticated user's identity as a UserContext token, and Kendra applies index-level security trimming so that no snippet the requesting user isn't authorized to see ever reaches the agent.
• Relevance boosting: Kendra supports both runtime and index-level boosting driven by document attributes. Teams can, for example, boost by recency using _last_updated_at, by business category, or by exact matches on custom fields such as Department or ProjectCode.
• Retrieve API versus Query API: When wiring Kendra into an agent's toolset, prefer the Retrieve API over the general-purpose Query API. Retrieve skips UI-oriented search metadata entirely and instead returns dense, passage-level excerpts that are purpose-built for feeding straight into an LLM's context window.

Section 4: Deterministic Guardrails, Tool Validation, and Circuit Breakers

Pre-Execution (Feedforward) and Post-Execution (Feedback) Controls

Autonomous agents need firm software-level boundaries to stop privilege escalation, malformed tool invocations, or infinite execution loops:

• Feedforward validation (pre-execution): Before a tool call ever reaches a downstream enterprise system, the runtime checks its arguments against strict Pydantic schemas — confirming required fields exist, numeric or enum values sit within allowed ranges, and the caller's authorization token actually permits that action.
• Feedback sensors (post-execution): When a tool errors out, the harness intercepts the failure deterministically rather than letting it crash the loop or forwarding a raw stack trace to the model. Instead, it reshapes the failure into a clean, structured message — for instance, Error: Database table 'users_v2' not found. Available tables: ['users', 'orders'] — giving the agent something actionable it can use to adjust its next move.

Sandboxing, Step Budgets, and Automated Circuit Breakers

• Isolated sandboxing: Any dynamically generated code — say, Python produced by a data-analysis agent — should run inside disposable, isolated sandboxes such as Docker containers or gVisor micro-VMs, with tightly locked-down outbound network access.
• Step and cost budgets: Each agent task should be capped by a hard ceiling on tool-call iterations (for example, no more than 10) as well as a maximum token spend. Crossing either limit should force the harness to halt execution and hand the task off to a human operator.
• Tool circuit breakers: If a backend API or database keeps failing across successive retry attempts, the harness should trip a circuit breaker, flagging that tool as UNAVAILABLE in the agent's tool catalog so the planning layer is pushed toward alternative paths instead of repeatedly hammering a broken dependency.

Section 5: Production Implementation Blueprint

The example below is a complete, runnable Python application illustrating a production-grade Supervisor Multi-Agent Architecture. It combines Pydantic-based tool validation, step budgeting, deterministic error handling, and a wrapper tool around Amazon Kendra retrieval.

import os
import json
import time
from typing import List, Dict, Any, Optional
from pydantic import BaseModel, Field, ValidationError

# =====================================================================
# 1. TOOL SCHEMAS & ENTERPRISE INTEGRATION CONTRACTS
# =====================================================================
class KendraRetrieveInput(BaseModel):
    """Input contract for the Amazon Kendra retrieval tool."""
    query_text: str = Field(..., description="The natural language query string to search across corporate documentation.")
    department_filter: Optional[str] = Field(None, description="Optional department metadata filter (e.g., 'Engineering', 'HR').")
    user_id: str = Field(..., description="Authenticated user ID used for native Kendra ACL security trimming.")
class DatabaseQueryInput(BaseModel):
    """Input contract for enterprise relational database lookups."""
    query_type: str = Field(..., description="Must be 'SELECT'. Data mutation operations are strictly prohibited.")
    table_name: str = Field(..., description="Target database table name.")
    limit: int = Field(default=5, ge=1, le=20, description="Number of records to return.")
# =====================================================================
# 2. MOCK ENTERPRISE SERVICES & KENDRA TOOL ENGINE
# =====================================================================
class MockAmazonKendraClient:
    """Simulates Amazon Kendra's high-density Retrieve API with ACL security trimming."""
    def __init__(self):
        self._mock_index = [
            {
                "doc_id": "KENDRA-DOC-001",
                "content": "Production deployment requires dual sign-off from Engineering and Security leads.",
                "department": "Engineering",
                "acl_users": ["user_eng_101", "admin_007"]
            },
            {
                "doc_id": "KENDRA-DOC-002",
                "content": "Standard employee travel stipend is capped at $150/day for domestic lodging.",
                "department": "HR",
                "acl_users": ["user_hr_201", "user_eng_101", "admin_007"]
            }
        ]
    def retrieve(self, query_text: str, user_id: str, department_filter: Optional[str] = None) -> List[Dict[str, Any]]:
        results = []
        for doc in self._mock_index:
            # Enforce Document-Level ACL Security Trimming
            if user_id not in doc["acl_users"]:
                continue
            # Apply Optional Metadata Department Filtering
            if department_filter and doc["department"].lower() != department_filter.lower():
                continue

            results.append({
                "DocumentId": doc["doc_id"],
                "ContentSnippet": doc["content"],
                "Department": doc["department"]
            })
        return results
class MockEnterpriseDatabase:
    """Simulates a secure internal enterprise database."""
    def __init__(self):
        self._tables = {
            "deployments": [
                {"id": 1, "service": "auth-service", "status": "COMPLETED", "env": "prod"},
                {"id": 2, "service": "payment-api", "status": "PENDING_APPROVAL", "env": "prod"}
            ]
        }
    def execute_select(self, query_type: str, table_name: str, limit: int) -> List[Dict[str, Any]]:
        if query_type.upper() != "SELECT":
            raise ValueError(f"Security Alert: Unauthorized operation '{query_type}'. Only 'SELECT' is permitted.")
        if table_name not in self._tables:
            raise KeyError(f"Database Error: Table '{table_name}' does not exist. Available tables: {list(self._tables.keys())}")
        return self._tables[table_name][:limit]
# =====================================================================
# 3. PRODUCTION AGENT HARNESS & RUNTIME ENGINE
# =====================================================================
class ProductionAgentRuntime:
    """
    Deterministic software harness surrounding probabilistic reasoning models.
    Enforces budgets, schema validation, sandboxing, and error recovery loops.
    """
    def __init__(self, user_id: str, max_step_budget: int = 4):
        self.user_id = user_id
        self.max_step_budget = max_step_budget
        self.kendra_service = MockAmazonKendraClient()
        self.db_service = MockEnterpriseDatabase()
    def execute_task(self, task_goal: str) -> Dict[str, Any]:
        print(f"=== [Harness Started] Initiating Task: '{task_goal}' for User: '{self.user_id}' ===")
        step_count = 0
        scratchpad_history: List[str] = []
        # Simulated dynamic model trajectory outputs (demonstrating multi-step execution & self-correction)
        simulated_llm_turns = [
            # Turn 1: Attempt invalid database deletion (Caught by Feedforward Schema/Guardrail)
            {
                "thought": "I will clean up old deployment logs before checking security compliance.",
                "action": "execute_db_query",
                "args": {"query_type": "DELETE", "table_name": "deployments", "limit": 5}
            },
            # Turn 2: Corrected database lookup
            {
                "thought": "I will check active deployment statuses in the enterprise database.",
                "action": "execute_db_query",
                "args": {"query_type": "SELECT", "table_name": "deployments", "limit": 2}
            },
            # Turn 3: Ground task using Amazon Kendra Retrieve API
            {
                "thought": "Now I need to query enterprise policies regarding production deployment sign-off.",
                "action": "kendra_retrieve",
                "args": {"query_text": "production deployment sign-off rules", "department_filter": "Engineering"}
            }
        ]
        while step_count < self.max_step_budget:
            step_count += 1
            print(f"\n--- [Step {step_count}/{self.max_step_budget}] ---")
            # Fetch current simulated model decision turn
            turn_data = simulated_llm_turns[min(step_count - 1, len(simulated_llm_turns) - 1)]
            print(f"Agent Thought: {turn_data['thought']}")

            action = turn_data.get("action")
            args = turn_data.get("args", {})
            # --- TOOL EXECUTION BRANCH: DATABASE ---
            if action == "execute_db_query":
                try:
                    # 1. Pre-execution Feedforward Schema Validation
                    validated_args = DatabaseQueryInput(**args)

                    # 2. Tool Execution
                    db_results = self.db_service.execute_select(
                        query_type=validated_args.query_type,
                        table_name=validated_args.table_name,
                        limit=validated_args.limit
                    )
                    observation = f"Database Query Success: {json.dumps(db_results)}"
                    print(f"[Observation]: {observation}")
                    scratchpad_history.append(observation)
                except ValidationError as ve:
                    error_msg = f"Schema Validation Blocked Action: {ve.errors()[0]['msg']}"
                    print(f"[Harness Feedforward Intercept]: {error_msg}")
                    scratchpad_history.append(error_msg)
                except (ValueError, KeyError) as exec_err:
                    error_msg = f"Tool Execution Failure: {str(exec_err)}"
                    print(f"[Harness Feedback Sensor Catch]: {error_msg}")
                    scratchpad_history.append(error_msg)
            # --- TOOL EXECUTION BRANCH: AMAZON KENDRA RETRIEVE ---
            elif action == "kendra_retrieve":
                try:
                    # Inject authenticated context
                    args["user_id"] = self.user_id

                    # 1. Pre-execution Schema Validation
                    validated_kendra_args = KendraRetrieveInput(**args)

                    # 2. Execute Kendra Retrieve Call
                    kendra_passages = self.kendra_service.retrieve(
                        query_text=validated_kendra_args.query_text,
                        user_id=validated_kendra_args.user_id,
                        department_filter=validated_kendra_args.department_filter
                    )

                    observation = f"Amazon Kendra Retrieved {len(kendra_passages)} Grounding Snippets: {json.dumps(kendra_passages)}"
                    print(f"[Observation]: {observation}")
                    scratchpad_history.append(observation)
                    # Successful multi-step completion condition reached
                    return {
                        "status": "SUCCESS",
                        "completed_in_steps": step_count,
                        "trajectory": scratchpad_history
                    }
                except ValidationError as ve:
                    error_msg = f"Kendra Schema Error: {ve.errors()[0]['msg']}"
                    print(f"[Harness Intercept]: {error_msg}")
                    scratchpad_history.append(error_msg)
        return {"status": "FAILED", "reason": "Step budget exhausted without completing task goals."}
# =====================================================================
# 4. EXECUTION DRIVER
# =====================================================================
if __name__ == "__main__":
    # Instantiate runtime for an authorized engineering user
    agent_runtime = ProductionAgentRuntime(user_id="user_eng_101", max_step_budget=4)

    # Run Agentic Workflow
    final_execution_summary = agent_runtime.execute_task(
        task_goal="Verify pending production deployments and confirm authorization policies."
    )

    print("\n================ FINAL SYSTEM SUMMARY ================")
    print(json.dumps(final_execution_summary, indent=2))

Section 6: Production Operations, Telemetry, and Governance

Observability: Tracing Trajectories with OpenTelemetry

Running agentic systems in production demands a level of tracing that ordinary APM dashboards cannot provide. Since agents follow variable, non-deterministic execution paths, your telemetry stack needs to reconstruct the entire trajectory tree an agent walked through, not just a single request-response pair:

• Span granularity: Each agent turn should emit a set of nested OpenTelemetry spans, with individual spans dedicated to building the system prompt, measuring how long the model takes to respond, checking that tool arguments match their expected schema, timing the actual tool call, and running any memory compaction that happens during that turn.
• Recording trajectory state: Spans need to log input context token counts, output token counts, the schemas used for tool parameters, and the raw length of returned observations. This level of detail is what lets you do accurate cost attribution and pinpoint exactly which step in a trajectory became a bottleneck.

Operational Guardrails and Evaluation (The Agentic Triad)

Keeping an agent healthy in production means running automated evaluation continuously across three key dimensions:

• Goal completion rate: the share of agent runs that reach a successful terminal state without running out of their step budget or throwing an unhandled tool exception.
• Context groundedness, or hallucination index: this checks whether the agent's final summarized answer is actually backed by facts pulled from its retrieval tools (such as passages returned by Amazon Kendra), rather than being generated from the model's own parametric memory.
• Tool execution precision: the ratio of well-formed, authorized tool calls versus calls that were malformed, failed schema validation, or attempted without proper authorization. A high rate of invalid calls is usually a sign that your prompt instructions are too loose or that your tool schemas have drifted out of sync with what the model expects.

Conclusion with Actionable Takeaways

Building enterprise-grade agentic AI systems means treating the underlying foundation model as a probabilistic reasoning component that lives inside a deterministic, tightly governed software harness. The organizations that successfully move agents from prototype into production are the ones that draw hard architectural lines between perception, planning, memory, and tool execution, rather than letting the model freely control all of them at once.

Actionable Blueprint for Architecture Teams

• Separate reasoning from execution: every LLM-initiated tool call should pass through a deterministic harness that validates arguments against a Pydantic schema before execution and catches errors cleanly afterward.
• Use Amazon Kendra for grounding: call Kendra's Retrieve API to feed the agent dense, passage-level context, while relying on its index-level document ACL trimming to keep security enforcement automatic.
• Cap memory usage explicitly: apply rolling context compaction and isolate sub-task context so that context rot, instruction drift, and runaway token spend don't creep in over long-running sessions.
• Set hard limits on execution: enforce strict caps on step count and token spend, and wire in tool-level circuit breakers so a single malfunctioning dependency can't trigger an infinite loop.
• Make trajectories observable: standardize on OpenTelemetry to trace every stage of the agent's reasoning loop, logging token usage, tool latencies, and full trajectory paths so you can run ongoing offline evaluation.