Practical notes: Smart Ways to Improve AI Agent Responses in Production

Use this as an operator-facing rebuild of the ideas in “Smart Ways to Improve AI Agent Responses in Production”: clear stages, ordered code slots, and recovery notes that survive a handoff. The Overview stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

1. Give the Agent the Right Context, Not More Context

For the 1 Give the Agent stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

User Question
      ↓
Query Embedding
      ↓
Vector Search
      ↓
Top-K Results
      ↓
Reranking
      ↓
Relevant Chunks
      ↓
LLM
      ↓
Final Answer

2. Debug Retrieval Before Changing the LLM

For the 2 Debug Retrieval Before stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Prefer structured outputs with schema validation over free-form prose when the next step is code or a tool call.

Question
   ↓
Query Transformation
   ↓
Retrieved Documents
   ↓
Similarity Scores
   ↓
Reranking
   ↓
Final Context
   ↓
Prompt
   ↓
LLM Response

results = vector_store.similarity_search(
    query=user_question,
    k=5
)

for result in results:
    print("Score:", result.score)
    print("Source:", result.metadata.get("source"))
    print("Content:", result.page_content[:500])

3. Improve Chunking Before Increasing Context

For the 3 Improve Chunking Before stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness. For the 3 Improve Chunking Before stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Chunk 1:
Employees are eligible for reimbursement when...

Chunk 2:
...the expense was approved by their manager and
submitted within 30 days.

4. Don’t Send the Entire Conversation to the Model

When working through the 4 Don t Send stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Cache stable system instructions and tool schemas. Re-sending identical preamble is a common source of burn.

Conversation Context

Recent Messages:
- User asked about refund eligibility.
- Agent explained the standard policy.
- User mentioned they purchased an annual plan.

Conversation Summary:
Customer purchased an annual subscription
and wants to know whether they qualify for a refund.

Current Question:
Can I still get a refund?

5. Separate System Instructions, Context, and User Input

When working through the 5 Separate System Instructions stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

SYSTEM INSTRUCTIONS

You are a customer support agent.
Answer using the provided knowledge.
Do not invent company policies.
If the answer isn't available, say that you don't know.
KNOWLEDGE
<retrieved_documents>
USER QUESTION
<user_question>

6. Teach the Agent When It Should Say “you Don’t Know”

When working through the 6 Teach the Agent stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node. When working through the 6 Teach the Agent stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

If the answer cannot be supported by the provided
knowledge, do not guess.

Clearly state that the information is unavailable.

7. Keep Business Logic Out of the Prompt

The 7 Keep Business Logic stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Budget tokens per turn and per session. Agentic tools expand context aggressively; hard caps keep demos from becoming surprise invoices.

def check_refund_eligibility(customer, purchase):
    if not customer.is_premium:
        return False

if customer.account_age_years < 2:
        return False
    if purchase.days_since_purchase > 30:
        return False
    if customer.previous_refund:
        return False
    return True

LLM
→ Understands the request
→ Decides which tool to use
→ Explains the result

Application
→ Enforces business rules
→ Validates data
→ Performs deterministic operations

8. Give Tools Clear Responsibilities

The 8 Give Tools Clear stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Expose tools with narrow schemas and explicit side-effect labels. Hosts need to know which calls mutate state before they auto-approve.

process_customer_data()

get_customer_order()
cancel_customer_order()
update_customer_address()
create_support_ticket()

9. Validate What the Agent Produces

The 9 Validate What the stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Keep graph state flat and typed. Nested blobs hide which node wrote which field and break resume after interrupts. The 9 Validate What the stage works best when treated as a measurable surface. Capture one golden transcript, one failure case, and the rollback note before expanding scope. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

User
 ↓
LLM
 ↓
Refund API

User
 ↓
LLM
 ↓
Tool Request
 ↓
Application Validation
 ↓
Business Rules
 ↓
Refund API

{
  "customer_id": "12345",
  "eligible": true,
  "reason": "Purchase is within the refund window"
}

10. Don’t Add Multiple Agents Without a Real Reason

For the 10 Don t Add stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

                    User
                     ↓
                Router Agent
                /     |     \
               ↓      ↓      ↓
             RAG     SQL     API
               \      |      /
                \     |     /
                 Final Agent
                     ↓
                  Response

11. Build an Evaluation Dataset From Real Questions

For the 11 Build an Evaluation stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness.

Easy questions
Ambiguous questions
Multi-step questions
Out-of-domain questions
No-answer questions
Tool-use questions
Adversarial questions

12. Trace the Entire Agent Workflow

For the 12 Trace the Entire stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Prefer small, testable units over sprawling scripts. When a step fails, the failure should point at a single responsibility rather than a tangled pipeline. Put human approval on edges that spend money or change production data. Compile-time wiring does not equal business completeness. For the 12 Trace the Entire stage, define the inputs, the owner of the step, and the exit criteria before changing code. Operators should be able to re-run the step from a known checkpoint without guessing hidden state. Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Request ID
    ↓
User Question
    ↓
Query Transformation
    ↓
Retrieved Documents
    ↓
Reranking Results
    ↓
Prompt Version
    ↓
Model
    ↓
Tool Calls
    ↓
Tool Responses
    ↓
Final Response
    ↓
Validation

13. Don’t Optimize Only for Accuracy

When working through the 13 Don t Optimize stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Keep configuration outside application code. Environment files, secret stores, and feature flags belong in one place operators can audit without reading the whole graph. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

Response Quality
      +
Reliability
      +
Latency
      +
Cost
      +
User Experience

Final Thoughts

When working through the Final Thoughts stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest. Document the happy path and the recovery path together. Retries, human gates, and dead-letter handling are part of the product, not later polish. Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

Operational checklist

When working through the Operational checklist stage, write down the contract first: required inputs, success signal, and what happens on partial failure. That checklist keeps later code changes honest.

Treat this stage as a contract between inputs and validated outputs. Name the artifacts, define success checks, and refuse silent partial completion.

Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

Pin dependency versions and record the image digest that ran the demo. Reproducibility beats tribal knowledge.

Record timings and token or query cost next to functional results. Cost visibility early prevents surprise bills when the path moves from demo to shared environments.

Checkpoint after expensive steps. Resume should not re-bill the same LLM call when an operator retries a later node.

Before promoting the stack, freeze versions, capture a golden transcript for the critical path, and confirm rollback steps. Shared environments need rate limits, tenancy checks, and a clear owner for secret rotation. Prefer boring reliability over clever one-off demos.

Batch note for 1481fc99429e: keep provider keys out of the repo, set a per-session token ceiling, and store transcripts next to the eval fixtures so later model swaps stay comparable.