AI Agent Foundations: Purpose, Prompt Design and Model Selection

A healthcare startup spent six months on an AI agent for patient triage. It ran on the newest GPT-4 model, the team included seasoned ML engineers, the interface looked great and investors loved the demo. Then it reached real clinics, and within days the nurses had quit using it. The agent asked questions that had nothing to do with the complaint, overlooked serious symptoms and gave inconsistent advice. Sometimes it was overcautious and sent patients with minor problems to the emergency room. At other times it brushed off conditions that needed attention.

The model wasn't to blame, and neither was the code. The foundation was shaky. The team hadn't pinned down what the agent was for, had hurried through prompt design and had picked the wrong model configuration. After three more months of rework, the agent now handles more than 10,000 patient interactions a day, with 94% nurse satisfaction and measurably better outcomes for patients.

Three foundation steps decide whether an agent succeeds: defining its purpose, designing its prompts and choosing its model. Get them right and the rest gets easier. Get them wrong and no engineering effort makes up for it.

Step 1: define the purpose

Before any code gets written, four questions need precise answers. What problem is being solved? Who will use the agent? What does success look like? What limits apply? Fuzzy answers produce failed projects, and specific ones produce systems that survive production.

When an agent is the right tool

An agent isn't the answer to every problem. Rule-based systems handle simple workflows better, classic ML models suit pure prediction, and human experts are the better choice when a high-stakes decision needs someone accountable.

Agents earn their keep when a job demands autonomous decisions across a complex, multi-step workflow with several information sources and tools.

Take fraud detection at a bank. A plain rule catches obvious patterns, like duplicate transactions or a card used in two distant places at once. Sophisticated fraud needs investigation: the agent has to analyze the transaction history, compare it with the customer's normal behavior, check merchant reputation databases, compute risk scores from multiple models, connect the case to known fraud networks and assemble an evidence package for a human reviewer. That calls for autonomy, since it runs around the clock over thousands of transactions, plus tool use across databases and APIs, reasoning that links weak signals and orchestration of a multi-step investigation. It suits an agent well.

Checking an account balance is the opposite case. It needs one API call, not an autonomous agent. The rule is to pick the simplest thing that works.

Who the users are

Users shape the architecture. A field technician wants mobile access that works offline, a data analyst wants dashboards with drill-down and an automated system wants clean APIs with predictable response formats. Some concrete cases:

• Retail inventory managers want short suggestions they can accept or override. A message such as "Reorder 150 units of Product X by Friday" can be acted on; a raw optimization score of 0.847 can't. The agent has to talk the way the user does and slot into the existing routine.
• Agents serving smallholder farmers in rural areas have to plan around poor connectivity. Text-message alerts may replace a web dashboard, the agent must tolerate an unreliable connection and its advice should come in local languages. A plain note to inspect field 3 for pests this afternoon helps far more than a sector anomaly flagged at 0.73 confidence.
• Clinicians and patients need different interfaces. A doctor reviewing recommendations wants the clinical reasoning, confidence scores and references to the literature, while a patient wants plain explanations and clear next steps. It's one agent behind two interfaces.

Success that can be measured

Goals like "improve efficiency" practically guarantee failure. Measurable targets are what drive decisions.

For a quality-control agent in manufacturing, success might be catching 95% of the defects human inspectors catch, keeping false positives below 10%, processing each item within 2 seconds, cutting the overall rate of escaped defects by 30% and holding 99.9% uptime during production hours. Those numbers influence the architecture directly. The 2-second limit rules out some models, the 95% detection rate needs plenty of training data and the uptime figure calls for redundancy and careful error handling.

An aviation maintenance agent is judged on other dimensions: predicting component failures 30 days ahead at 80% accuracy, reducing unscheduled maintenance by 40%, achieving 100% regulatory compliance in recommendations, never advising that safety-critical work be postponed and producing a full audit trail for every decision. The compliance target means rigorous testing and human oversight, and the safety rule means conservative decision thresholds. These are architectural requirements and not post-launch tuning.

Constraints

Every production system runs into limits of three kinds. Technical ones include latency targets, infrastructure restrictions and integration needs. Business ones include budget, timeline and what the organization can actually support. Regulatory ones cover data privacy, industry compliance and audit requirements.

• Banking is dominated by regulation. A fraud agent keeps complete audit trails, explains every decision, never touches customer data without authorization and follows rules that differ by jurisdiction. None of that is negotiable.
• Healthcare agents must meet HIPAA. Patient data is encrypted in storage and in transit, access is logged and auditable, retention follows set rules and some information can't be stored even briefly. These limits bear directly on memory design and deployment options.
• Retail agents deal with different pressures. Holiday peaks require horizontal scaling, legacy point-of-sale integration narrows technology choices and cost sensitivity pushes model choice toward cheaper options. All of this shapes the whole stack.

Write the constraints down. They define what's possible.

Five worked examples

Each of these captures the purpose, users, success measures and constraints together.

• Banking, a transaction monitoring agent. It detects and investigates suspicious transactions live across millions per day, for fraud analysts and compliance officers. Success means catching 90% of fraud with under 5% false positives, working each case within 30 seconds and keeping a complete audit trail. Limits include regulatory compliance, explaining every decision, never blocking a legitimate transaction without analyst review and 99.95% uptime around the clock.
• Retail, a dynamic pricing agent. It tunes prices for thousands of SKUs using demand, competition, stock levels and margin targets, for category managers and pricing analysts. Success is 8% more revenue at target margins, prices refreshed twice a day, seasonal swings handled and price floors and ceilings respected. Limits include the existing ERP integration, never pricing below cost, staying competitive and following pricing rules.
• Healthcare, a patient triage agent. It assesses symptoms, sets urgency, recommends a care setting and books appointments, for nurses, patients and clinic administrators. Success is 95% agreement with nurse assessments, 30% shorter waits, better satisfaction and immediate routing of urgent cases. Limits include HIPAA, never delaying emergency care, empathetic communication, EHR integration and multiple languages.
• Manufacturing, a predictive maintenance agent. It watches equipment health, predicts failures, schedules preventive work and allocates crews, for maintenance supervisors, plant managers and technicians. Success is predicting failures 30 days ahead at 85% accuracy, 50% less unplanned downtime, lower maintenance costs and equipment lasting 20% longer. Limits include real-time sensor processing, the existing CMMS, legacy equipment, safety compliance and offline operation.
• Agriculture, a crop health agent. It monitors crops with satellite imagery and ground sensors, spots disease outbreaks, recommends interventions and tunes irrigation and fertilization, for farm managers, agronomists and cooperative managers. Success is detecting disease a week earlier than manual inspection, 25% less water, 15% higher yield and recommendations in the local language. Limits include patchy connectivity, low-end phones, existing farm software, SMS alerts and many crop types.

Every use case has its own requirements, and they shape the whole architecture, so this step deserves real time.

Step 2: design the prompts

The system prompt is the agent's operating manual. It sets personality, capabilities, limits and behavior patterns. A well-built one makes behavior consistent, safe and reliable, and a poor one gives unpredictable results that fall apart in production.

Goals

Begin with what the agent should accomplish, and be specific and complete about it.

A banking fraud analyst agent might be told to analyze transactions for fraud indicators, score risk with the supplied scoring models, cross-check against known fraud patterns, write detailed investigation reports and escalate high-risk cases at once. It's also told to stay professional in all communication, never reveal investigation details to customers and always give reasoning for its risk assessments. Together these set the scope and the boundaries, so the agent knows what to do and what to stay away from.

A retail customer service agent gets a different list. It resolves questions about orders, products and policies, looks up order history and inventory, processes returns inside policy, hands refund requests above $500 to a human, keeps a friendly tone and gathers feedback after each interaction. It's barred from promising anything outside company policy, and it should offer alternatives when it can't grant a request. The boundaries are explicit: it can't approve large refunds, it must escalate them and it must stay within policy. Those limits head off problem behavior.

Persona

The persona changes how consistent the agent feels and how much users trust it. A triage agent in healthcare should sound calm and professional, which reassures people. A tutoring agent should be encouraging and patient to keep learners motivated, and a technical support agent should be knowledgeable and methodical so that its fixes feel dependable.

For aviation maintenance, the persona is an experienced aircraft maintenance engineer with twenty years across several aircraft types. Safety outranks everything, manufacturer guidelines are followed strictly, communication is clear and technically precise, decisions are never rushed when safety is involved and everything is documented for regulators. That persona shapes how the agent approaches problems, reports findings and decides, and it keeps thousands of interactions consistent.

An agricultural advisor for smallholder farmers calls for another voice. It knows local crops, climate and farming practice, speaks in simple practical terms, understands the resource limits farmers face, offers advice that fits the tools and budget on hand, respects traditional knowledge while introducing modern methods and stays patient and encouraging.

The persona has to match the audience. Technical jargon is fine for experts, plain language suits general audiences and cultural sensitivity matters in global deployments.

Instructions

Instructions tell the agent how to perform tasks. Loose ones produce uneven behavior, and specific ones produce reliability. Numbered procedures work well, and these two examples show the style.

A quality-inspection agent looking at product images follows eight steps:

1. Load the image and pull out features with the vision model.
2. Compare them against the quality spec for that product type.
3. Find any defects and mark them with bounding boxes and a class.
4. Score each defect's severity with the supplied rubric.
5. Reject the product and log the details if severity passes the threshold.
6. Ask for human review with supporting evidence if severity is borderline.
7. Update the quality metrics in the database.
8. Alert supervisors immediately when several defects of one type show up in a row.

Steps like these cut ambiguity, and the agent knows what to do in each situation.

A patient-assessment agent follows ten:

1. Gather symptoms, how long they've lasted and how severe they are, through the structured questionnaire.
2. Ask follow-ups depending on the symptom cluster.
3. Check for red flags that need immediate care.
4. Classify as an emergency and send to the immediate pathway if any red flag appears.
5. Compute a triage score with the approved algorithm.
6. Recommend a care setting from the score and symptoms.
7. Book an appointment when one applies.
8. Give the patient clear next steps and a timeline.
9. Never diagnose a condition or recommend a specific treatment.
10. Lean toward caution whenever a case is ambiguous.

The "never diagnose" line matters most here, since it draws a hard boundary that prevents liability problems.

Guardrails

Guardrails block harmful or problematic behavior, and production systems can't do without them. There are three families:

• Content guardrails prevent inappropriate output. A customer service agent never swears even when the customer does, and a financial advisory agent never guarantees returns or makes promises about the market.
• Safety guardrails prevent dangerous actions. An aviation agent never recommends postponing safety-critical maintenance, a healthcare agent never suggests delaying emergency care and a manufacturing agent never disables safety interlocks.
• Privacy guardrails prevent leaks. A banking agent never shares one customer's information with another account, a healthcare agent never discloses patient data without authorization and a retail agent never exposes sensitive pricing algorithms.

A fraud detection agent at a bank might carry these rules. Never reveal the detection methods or models to any user. Never share investigation details with an account holder under investigation. Never open an account without a legitimate fraud-related reason. Never decide on an account closure without analyst approval. Never carry customer information from one investigation to another. Always keep audit logs of account access. When unsure about regulatory compliance, escalate to a human supervisor. These protect the bank, its customers and the integrity of the investigations.

A healthcare triage agent would carry others. No specific diagnoses or treatment advice. Immediate escalation for chest pain, trouble breathing, severe bleeding or other emergency symptoms. No suggestion to postpone care for serious symptoms to save money. No access to a patient's record without an active interaction. A consistently empathetic, professional tone. An immediate connection to crisis resources if a patient expresses suicidal thoughts. No sharing of patient information with unauthorized parties. A lean toward higher acuity whenever urgency is unclear. These protect patient safety and make sure care is appropriate.

Testing and iterating

The first prompt won't be perfect. Try it on real scenarios, hunt for edge cases and refine in rounds. A workable loop is to start from a basic prompt, run 20 to 30 representative scenarios, record the failures, look for patterns, revise the prompt and repeat.

A retail pricing agent may turn out to recommend prices below cost during high demand. The fix is an explicit rule against pricing below cost plus a 10% minimum margin. A triage agent may wave off recurring headaches as minor when they could signal something serious, and the fix is an instruction to always recommend a medical evaluation for symptoms lasting over two weeks, whatever their apparent severity.

Testing exposes gaps in instructions, missing guardrails and ambiguous guidance. This refinement isn't optional. It's how reliable production systems get built.

Complete prompt examples

Putting the pieces together, every prompt has the same skeleton of role, goals, instructions and guardrails, adapted to its domain.

The banking fraud analyst takes the role of a fraud specialist with fifteen years in financial crime investigation, producing detailed risk assessments for analysts. Its goals are to look for anomalies in transaction patterns, calculate risk scores with approved models, check transactions against known fraud databases, write thorough reports with evidence and send high-risk cases straight to human analysts. When analyzing a transaction it proceeds as follows:

1. Pull the account's full transaction history.
2. Look for velocity anomalies, meaning unusual frequency or amounts.
3. Verify that merchant and location fit the customer's profile.
4. Estimate fraud probability with the ensemble model.
5. Escalate immediately with detailed evidence if the score is above 0.8.
6. Flag for analyst review with supporting data if it falls between 0.5 and 0.8.
7. Clear the transaction and log the decision if it's below 0.5.
8. Record the reasoning and data sources every time.

Its guardrails: reveal no detection methods, discuss no ongoing investigation with account holders, access no account without a legitimate reason, keep an audit trail of every decision, escalate regulatory doubts to a supervisor and approve no account closure without analyst confirmation.

The retail customer service agent plays a helpful, friendly representative for a premium brand who assists with orders, products and policies while keeping the experience excellent. Its goals are fast and effective resolution, returns and exchanges within policy, accurate product information and recommendations, a consistent brand voice and appropriate escalation of complex issues. For order inquiries it:

1. Looks up the order by number or email.
2. Gives the current status and expected delivery date.
3. Explains the reason and offers alternatives if the order is late.
4. Handles tracking updates and any worries.
5. Escalates to the fulfillment team if the order is missing.

For returns it:

1. Confirms the purchase falls within the 30-day window.
2. Checks that the product's condition meets the criteria.
3. Issues a return authorization and a label.
4. Starts the refund once the item arrives.
5. Sends refunds above $500 to a manager for approval.

Its guardrails: no refund outside policy without manager approval, no promises about delivery dates it can't control, no disclosure of supplier or cost information, a professional and empathetic tone even with frustrated customers, a polite exit and escalation if a customer turns abusive and no return without proper verification.

The healthcare triage agent is a compassionate, experienced triage nurse who helps patients find the right care setting by assessing symptoms and urgency while keeping them safe. Its goals are accurate symptom and urgency assessment, routing to emergency, urgent, primary or self-care, immediate attention for emergencies, fewer unnecessary emergency visits and sustained patient trust. During an assessment it:

1. Records the chief complaint and symptom details through structured questions.
2. Asks about duration, severity and progression.
3. Checks for red flags that need immediate care.
4. Screens for vital sign abnormalities when they're available.
5. Computes triage acuity with the approved algorithm.
6. Gives a recommendation with clear reasoning.
7. Books an appointment where appropriate.
8. Spells out the next steps.
9. Adds return precautions describing when to seek a higher level of care.

Red flags that mean emergency care are chest pain or pressure, difficulty breathing, severe bleeding or trauma, a sudden severe headache or vision changes, confusion or an altered mental state, signs of stroke such as a drooping face, weak arm or trouble speaking, a severe allergic reaction, and suicidal or homicidal thoughts. Its guardrails: no specific diagnoses or treatment advice, red flags always treated as emergencies regardless of other factors, no suggestion to delay emergency care for cost or convenience, empathetic and patient-centered communication, crisis resources right away if suicidal thoughts come up, no access to records without active patient consent, a lean toward higher acuity when urgency is uncertain and escalation of complex cases to a nurse supervisor.

Each prompt fits its own domain, user type and risk profile. The banking agent centers on investigation and compliance, the retail agent balances customer satisfaction against policy and the healthcare agent puts safety ahead of everything.

Step 3: choose the model

Model choice affects performance, cost, latency and capability directly, and a wrong pick undermines everything else.

Base model options

The market shifts quickly, but the core trade-offs stay put: capability against cost, latency against quality, hosted against self-hosted and general-purpose against specialized.

• GPT-4 and GPT-4 Turbo from OpenAI reason very well, follow instructions closely, write good creative content and call functions reliably, with broad tool-use support. The downsides are higher cost, moderate latency and a dependency on OpenAI's API. They fit complex reasoning, creative work, general-purpose agents and cases where quality matters more than price.
• Claude 3.5 and Claude 4.5 from Anthropic stand out at long-document analysis, reason strongly, generate excellent code, follow subtle instructions and offer a 200K-token context window with strong safety alignment. They cost more, come only through an API and have rate limits on the free tier. They suit document analysis, research agents, coding assistants, long-context applications and cases where safety and alignment matter.
• Llama 3 70B and 405B from Meta are open source and self-hostable with no API fees, perform well, have strong community support and come in many fine-tuned variants. You need infrastructure to host them, pay for inference and need the expertise to deploy them, and they trail the frontier models. They fit on-premise deployments, data privacy needs, high-volume use where API costs would be prohibitive and heavy customization.
• Gemini Pro and Ultra from Google handle multiple modalities, reason well, integrate with Google services and are competitively priced. They're less proven in production agent systems and have a smaller ecosystem. They suit vision applications, Google Cloud integration and multimodal agents.
• Domain-specific models exist for healthcare (Med-PaLM), code (CodeLlama, StarCoder), finance and other fields. They do better on narrow use cases and lack general capability.

Tuning parameters for production

Temperature controls randomness. Low values from 0.0 to 0.3 give consistent, deterministic output, while high values from 0.7 to 1.0 give creative, varied output. A banking fraud agent should run at 0.0, since consistency beats creativity and the same transaction ought to get the same risk assessment every time. A customer service agent works at 0.3 to 0.5, which allows some natural variation in phrasing while policy interpretation stays steady. Agents that write marketing copy or product descriptions do well at 0.7 to 0.9, where variety adds value.

Top-p, or nucleus sampling, is a second randomness control. Low values from 0.1 to 0.5 restrict choices to the most likely tokens, and higher values from 0.7 to 0.95 allow more diversity.

Max tokens caps the output length and should match the use case. A fraud investigation report needs 1000 to 2000 tokens, a customer service reply 200 to 500 and a simple API call response 50 to 100.

Tune with production data and not demo scenarios. Use real user input, measure quality and iterate.

Context window

The context window sets how much the agent can weigh at once. Longer windows support more elaborate reasoning but add cost and latency. Window sizes differ widely: 128K tokens for GPT-4 Turbo, 200K for Claude 3.5, 8K for Llama 3 70B (up to 32K in some variants) and as much as 1M for Gemini Pro.

A healthcare triage agent is fine with 8K, because each interaction is a few hundred tokens of conversation plus a few pages of clinical guidelines. A legal contract analysis agent benefits from 200K, which fits entire contracts with supporting case law and regulations in one window and improves quality a lot compared with chunking documents. A customer service agent with product catalogs, order history and policy documents sits comfortably at 32K to 64K, a good balance of capability and cost.

The memory architecture feeds into this choice as well. Vector databases can retrieve the relevant information, which reduces the need for a huge window.

Cost, latency and accuracy

Every production system trades these three off against each other.

On cost, GPT-4 is priced near $30 per million input tokens and $60 per million output tokens, with Claude 3.5 Sonnet in the same range, while Llama 3 70B brings infrastructure costs and no per-token fee. A busy retail service agent handling 100,000 interactions daily at around 500 tokens each could reach $1,500 a day in API costs, or $45,000 a month, which may justify self-hosting Llama or using smaller models. A quiet aviation maintenance agent handling 1,000 complex interactions monthly might cost about $500, and GPT-4 quality is a sensible purchase there.

On latency, first-token time runs from 200 milliseconds to 2 seconds depending on model size and hosting, and total response time adds inference, tool calls and processing. Real-time customer chat feels slow above 3 seconds, so use faster models or streaming. Batch fraud investigations can take 10 to 30 seconds per transaction if accuracy improves. A quality-control agent on an assembly line must answer in under 2 seconds, so it needs optimized deployment or smaller models.

On accuracy, larger models generally reason better and follow instructions more closely, and the gap between GPT-4 and smaller models can be measured. Where decisions carry high stakes, as in healthcare triage, financial fraud and safety-critical systems, buy the best model, because a mistake costs more than the model does. Lower-stakes jobs such as product recommendations, simple service and content categorization are often fine with smaller models. Test on production data to see the actual quality difference for the case at hand, since impressions often differ from reality.

Combining several models

Production systems frequently use different models for different tasks, and requests get routed to whichever fits. Cheap, fast models take simple tasks and expensive, powerful ones take complex tasks. Measure cost and quality on your own workload.

• At a bank, fraud work could split three ways: GPT-4 for the deep pattern analysis and write-ups (high quality, few calls), GPT-3.5 Turbo for the many customer notifications (good enough and cheaper) and a fine-tuned Llama 3 8B to classify transactions quickly and cheaply.
• A healthcare triage system could use Claude 3.5 for symptom analysis that needs long medical context, GPT-4 for natural, empathetic patient communication, and a specialized medical model for particular clinical decision support.
• Retail could rely on GPT-4 for service conversations that require reasoning, embedding models behind product search and recommendations, and a fine-tuned classifier for simple routing.

Model choices by industry

• Banking fraud detection: Claude 3.5 as the primary, because its 200K context holds a full transaction history plus regulatory guidance, and GPT-4 Turbo as secondary for customer communications. Investigations need deep pattern analysis over long histories and regulatory documents, which justifies the higher cost.
• Retail inventory optimization: GPT-4 Turbo as primary for demand-forecasting reasoning, with self-hosted Llama 3 70B as secondary for real-time pricing. Strategy needs quality reasoning, while real-time pricing runs at volume where self-hosting pays off.
• Healthcare patient triage: GPT-4 as primary for proven reliability on safety-critical decisions, Claude 3.5 as fallback for complex cases with long medical histories and a fine-tuned domain model as secondary for routine classification. Complex cases need the best quality, and routine ones can use cheaper specialists.
• Manufacturing quality control: GPT-4 Vision as primary for image analysis, plus a fine-tuned Llama 3 8B on premises for fast defect classification. Assembly-line inspection needs low latency, the small model covers routine cases and GPT-4 Vision takes the escalated visual ones.
• Agriculture crop monitoring: self-hosted Llama 3 70B on edge devices as primary, with GPT-4 as secondary for complex disease diagnosis from images. Spotty connectivity and remote sites favor local hosting, routine monitoring stays local and hard cases connect to the cloud.
• Aviation maintenance: GPT-4 as primary because safety-critical decisions need the best quality, with Claude 3.5 as backup for reading maintenance manuals and regulations. Safety isn't negotiable, regulations mean long documents and redundancy plus human oversight are mandatory.

Selection has to reflect your own accuracy needs, latency limits, volume economics, privacy requirements, deployment environment and risk tolerance.

Putting the foundation together

The foundation determines everything that follows. Define the purpose clearly with measurable success criteria, design prompts that produce consistent, safe behavior and choose models that balance quality, cost and latency for the use case at hand.

These three steps account for something like 60% of an agent's success. Skipping or rushing them produces systems that fail in production, while doing them well makes the remaining architectural pieces straightforward.