This article is published in English.
Right-Sizing LLMs: Routing, Retrieval and Evaluation Over Raw Model Size
Learn how to choose between small and large language models by workload, measure cost per successful task, and use routing, RAG, caching and validation first.
Parameter counts make for easy headlines, and it is tempting to treat them as a proxy for product quality: a 70B model must beat a 7B one, so the largest model you can afford must be the safe choice. In production that shortcut breaks quickly. Larger models usually cost more per call, respond more slowly, add infrastructure weight and often solve problems your application never had. This guide shows how to choose a model by workload rather than by size, how to reason about cost, latency and failure together, and which architectural levers (routing, retrieval, validation, caching and plain code) usually deliver more than an upgrade.
Why "bigger is better" stops working in production
The intuition is understandable. Software engineers have spent decades watching hardware upgrades pay off: a faster CPU, more RAM, bigger disks and a newer GPU are almost always improvements. When language models grew larger and more capable, it felt natural to carry that mental model over. If one model reasons better than another, why would anyone deliberately pick the weaker one?
The answer appears the moment a model sits behind a real feature. The question you ask shifts from "which model is the smartest?" to "which model produces the best outcome for this specific job?" Those are very different questions. The largest candidate may give a marginally better answer while being far slower. It may cost several times as much. It may be pointless for a simple classification step, produce longer answers than your UI can display, burn through your context budget, and complicate your deployment. Most importantly, it may be solving a problem you do not actually have.
Start with the workload, not the parameter count
A more reliable way to select a model begins with a description of the work itself. Before comparing any models, answer these questions:
- What does the input look like?
- What output is expected?
- How hard is the reasoning involved?
- How much context does each request need?
- How much error can the feature tolerate?
- How fast must the response arrive?
- What is the acceptable cost per request?
- Does the feature need text generation at all?
- Could a smaller model, deterministic code, retrieval, caching or a mix of them do the job more efficiently?
That last question is the real engineering question, and the rest of this guide is about answering it: why the biggest model is not automatically the best one, how to decide between small and large, when a large model genuinely earns its cost, and what a sensible production design looks like.
One support product, two completely different workloads
Consider an enterprise support application. A user types "Reset my password." What should the AI layer do here? Most likely just recognize the intent and map it to a label such as the one below, so that the application can hand off to a fixed, deterministic workflow.
PASSWORD_RESET
No heavyweight reasoning model is needed to reach that label. Routing this request through one would be a questionable design choice, since a lightweight model, or even keyword and rule matching, could plausibly handle it.
Now imagine a different message: European customers have been seeing intermittent payment failures since yesterday's deployment, and the user wants the system to compare the deployment diff with payment-service logs, find likely failure patterns, judge whether the new retry mechanism is involved, and suggest a rollback plan. This request needs a lot more:
- retrieval of the relevant deployment changes and logs
- a large amount of context
- the ability to read and understand code
- analysis of log output
- reasoning that spans several steps
- correlation of events across systems
- a clear technical explanation
- honest handling of uncertainty
Here a stronger model can genuinely add value. The mistake is not using a large model; it is treating these two requests as if they were the same workload.
A working definition of model value
A useful way to frame the choice is a rough heuristic:
Model value = capability × reliability × usefulness ÷ cost
This is not a formula you compute. It is a way of thinking. A model that is 10% more capable but five times more expensive and three times slower is not automatically the better production option. Equally, a model that is extremely cheap but unreliable for your task is a poor choice too. What you are optimizing for is not peak intelligence but the most useful intelligence per unit of cost, latency and complexity.
Why large models are tempting, and where the constraints appear
Preferring big models is not irrational, because they bring real strengths. They often cope better with complex reasoning, perform more evenly across varied tasks, interpret vague instructions more gracefully, navigate complicated codebases more effectively, need less task-specific prompting, and can be far more capable on genuinely hard work.
So why not use the strongest model everywhere? Because production introduces constraints that leaderboard numbers rarely surface. Picture an endpoint handling 100,000 requests a day. If the larger model costs noticeably more per call, that gap is no longer abstract; it shows up in the infrastructure budget. If the larger model is also slower, users notice. If it tends toward verbose answers, output token spend climbs. And if the application runs thousands of tiny tasks, sending each one to an advanced reasoning model is simply waste.
Selecting a model is an optimization problem, not a popularity contest.
The model at the top of a benchmark table is not necessarily the one that produces the best application.
Parameter count is only one axis
Comparisons between models tend to fixate on size: 7B, 13B, 34B, 70B, hundreds of billions. Size alone tells you little about fit. A practical comparison looks at several dimensions at once, for example:
- capability on your specific task
- latency, both time to first token and total generation time
- cost per request and per successful task
- throughput under your expected load
- the context size the task actually needs
- reliability and consistency across repeated runs
- deployment options, including local or private hosting
- controllability
Controllability deserves special attention because it is easy to overlook. A model can be highly capable yet hard to keep within bounds. In enterprise workflows, predictable behaviour is often worth more than creativity.
Structured extraction is a different target
Take invoice extraction. The feature needs a fixed shape like the one below, not a thoughtful essay about the document.
{
"invoiceNumber": "...",
"invoiceDate": "...",
"vendor": "...",
"total": 0
}
What matters here is dependable, well-formed output that downstream code can parse every time. That is a separate optimization target from general reasoning ability, and smaller or more constrained models frequently meet it well, especially when combined with schema validation.
Latency: the first production trap
In a demo, a six-second wait feels fine. The answer is impressive, you share it with the team, and everyone is pleased. Put the same call behind a button in a real application and the experience changes: the user clicks, a spinner appears, and three, five, eight seconds pass. At that point nobody is admiring the model's intelligence. They are wondering why the application is so slow.
Latency is a product feature, and in interactive software it matters enormously.
Why bigger models tend to respond more slowly
The exact relationship depends on many factors: model architecture, hardware, the serving stack, quantization, batching, how many tokens are generated, prompt size and model design. As a general tendency, though, more computationally demanding models need more resources per token and can return results more slowly. That matters most in:
- chat interfaces
- coding assistants and autocomplete
- voice assistants
- customer support tools
- interactive dashboards
- agentic workflows, where waits compound across steps
Autocomplete makes the point sharply. A code suggestion that takes five seconds is no longer autocomplete; it is an interruption. A smaller model that answers almost instantly is often far more useful than a stronger one that makes the developer wait.
Streaming improves perception, not compute
Streaming is the standard way to make waits feel shorter. Without it, the user sees nothing until the full response is ready:
[wait...]
Hello! Here is the answer...
With streaming, the text grows on screen as tokens arrive:
Hello
Hello, here
Hello, here is
Hello, here is the
Hello, here is the answer...
Keep the distinction clear. Streaming reduces perceived latency because the first words appear quickly, but it does not reduce the compute spent or the total time until the answer is complete. It is a user-experience improvement, not a performance optimization, and it does nothing for non-interactive steps such as a classifier inside a pipeline.
Cost: measure it per successful task
Many prototypes turn into expensive systems at exactly this point. During development, spend looks negligible: one engineer sends a few prompts and nobody thinks about the bill. Then real traffic arrives, and each request can carry much more than the user's message:
- many concurrent users
- several requests per session
- long prompts
- retrieved documents
- tool results
- conversation history
- generated responses
Token volume grows fast, and the choice of model starts to matter a great deal.
A more honest metric than price per call is the cost of getting the task done correctly. Compare two hypothetical models (the figures are illustrative, not benchmark results):
- Model A: $0.01 per request, 90% task accuracy, about 1.11 requests needed per success, roughly $0.011 per successful task.
- Model B: $0.05 per request, 96% task accuracy, about 1.04 requests needed per success, roughly $0.052 per successful task.
The expected number of attempts is simply one divided by the success rate, so the cost per success is the request price divided by accuracy. If Model B costs five times as much while improving outcomes only slightly, the business may reasonably prefer Model A. The picture changes when a wrong answer is costly, for example when it triggers a refund, a compliance issue or an outage. That is why cost always has to be weighed together with the consequence of failure, not on its own.
Oversized models for undersized problems
A common anti-pattern looks like this: every incoming message is classified as a complaint, a question or a refund request, and every one of them goes to the most sophisticated model available. The reason is convenience: one API, one prompt, one model, done. But architecture is not about making one component capable of everything; it is about matching each job with a suitable component. For plain classification, options include:
- deterministic rules
- embeddings
- a small language model
- a dedicated classifier
- a larger model reserved for low-confidence cases
The last option leads to one of the most effective production patterns.
Escalation: the expensive model as an exception handler
The naive design sends everything straight to the large model:
Every request
↓
Large model
An escalation design lets a cheap model try first and checks how confident it is:
Every request
↓
Small/cheap model
↓
Confidence check
↓
┌───────────────┐
│ │
High confidence Low confidence
│ │
Fast answer Large model
High-confidence answers return immediately; only uncertain cases move up to the larger model. The expensive model stops being the default path and becomes an exception handler. The pattern depends on having a trustworthy confidence signal, such as a calibrated classifier score, agreement between methods, or a validator that can reject malformed output, so measure how often low-quality answers slip through the "high confidence" branch before relying on it.
Routing requests across tiers of models
Once you think this way, models stop looking like competitors and start looking like specialized workers. A request router can sit in front of several tiers and pick the right one for each request, with a shared validation step before anything reaches the user:
User Request
|
v
Request Router
|
+-----------+-----------+
| | |
v v v
Simple Medium Complex
| | |
v v v
Small LLM Mid Model Large Model
| | |
+-----------+-----------+
|
v
Validation Layer
|
v
Response
The router needs some notion of complexity. The simplest version is a small set of categories:
SIMPLE
MEDIUM
COMPLEX
The dispatching logic can then be as plain as a switch on the classifier's verdict. The example below is written in C#, but the language is incidental; the same structure fits in a TypeScript service just as well.
public async Task<string> ProcessAsync(Request request)
{
var complexity = await classifier.ClassifyAsync(request);
return complexity switch
{
Complexity.Simple =>
await smallModel.GenerateAsync(request),
Complexity.Medium =>
await mediumModel.GenerateAsync(request),
Complexity.Complex =>
await largeModel.GenerateAsync(request),
_ => throw new InvalidOperationException()
};
}
What matters is the architectural statement the code makes: not every request deserves maximum intelligence. Applied consistently, that one decision can change the economics of an AI system dramatically. Note that the classifier itself adds a call and some latency to every request, so it should be much cheaper than the models it routes to, and an unknown category should fail loudly, as the default branch does here.
When the gap is knowledge, not intelligence
Another frequent reflex is: "The model doesn't know our internal documentation, so let's switch to a bigger one." Size does not fix missing knowledge. When the information is proprietary, recent or highly domain-specific, the problem is access to knowledge rather than reasoning power. That is the case Retrieval-Augmented Generation (RAG) addresses. The basic flow looks like this:
User Question
|
v
Embedding / Retrieval
|
v
Relevant Documents
|
v
Prompt + Retrieved Context
|
v
Language Model
|
v
Answer
If a user asks about your company's internal refund policy for enterprise customers, no general-purpose model, however large, knows the answer. You have to supply the relevant text in the prompt. For a deeper look at how this retrieval step works, see our guide to how RAG systems retrieve fresh knowledge on demand.
Fix the information pipeline before the model
This leads to a principle worth adopting as a rule:
Improve what you feed the model before you upgrade the model.
Teams regularly try to fix poor answers by jumping to a larger model when the real cause lies elsewhere:
- weak retrieval
- irrelevant documents
- missing metadata
- poor chunking
- insufficient context
- stale information
- ambiguous instructions
In those cases the model is not the bottleneck; the information pipeline is.
Context quality beats context quantity
Large context windows are impressive, but more context is not automatically better. Handing a model 200 pages of documentation when two paragraphs contain the answer technically gives it the information and practically makes the task harder, because it now has to find the signal amid the noise. Oversized context tends to raise:
- token consumption
- latency
- cost
- distraction
- the chance of conflicting information
A better target is the smallest amount of high-quality context that still lets the model answer correctly. That is why mature RAG systems invest so heavily in the retrieval layer, through techniques such as:
- semantic and hybrid search
- filtering on metadata
- rewriting the user's query
- reranking candidate passages
- keeping documents fresh
- producing well-formed chunks
Hallucinations call for verification, not a bigger model
An uncomfortable truth: using a sufficiently powerful model does not make hallucinations disappear. Stronger models do tend to get more facts right and reason better across a range of tasks, yet a language model is still a text generator, never a source of truth you can query like a database. So the fix belongs in the design: add explicit verification to the flow, for example:
User Request
↓
Retrieve Evidence
↓
Generate Answer
↓
Validate Claims
↓
Return Response
For high-value workflows, you can strengthen that with:
- citations that point to evidence
- structured outputs checked against a schema
- validation against business rules
- lookups in the system of record and other tool calls
- calculations done in code
- human sign-off for the riskiest actions
Let the model reason and let software enforce the rules
This creates an important boundary in the architecture. If a model is asked to compute an invoice total, there is no reason to trust its arithmetic when your application can do the math exactly. Split the responsibilities instead:
Model:
Extract line items
Application:
Calculate subtotal
Application:
Calculate tax
Application:
Calculate total
Model:
Explain the result
The model extracts line items and explains the result; the application calculates the subtotal, tax and total. Each part does what it is reliable at, and the numbers the user sees are always correct by construction.
Where small models shine, and where they fall short
Small language models are often written off as "less intelligent," which is technically true in many settings. But engineering is not about intelligence in isolation, and small models bring concrete advantages:
- lower inference cost
- lower latency
- simpler local deployment
- lighter infrastructure requirements
- potentially higher throughput
- easier scaling
- a good fit for narrow tasks
- usefulness in edge scenarios
- potentially better privacy when run locally
They are especially attractive for classification, extraction, summarization, routing, autocomplete, simple transformations and domain-specific workflows. If you want to go further into this trend, our overview of small specialized models outperforming giant LLMs covers it in more depth.
Small does not mean automatically better, though. Some tasks exceed what a small model can do reliably: sophisticated reasoning across many sources, complex code analysis, hard planning problems or nuanced interpretation. There, a stronger model can justify its cost. Both extremes are bad advice. "Always use the biggest" wastes money, and "always use the smallest" ships poor quality. The better rule:
Use the smallest model that meets your application's quality threshold.
Workloads that justify a large model
None of this is an argument against large models. They are extremely useful, and certain workloads clearly repay the extra capability.
Complex multi-step reasoning
When a feature depends on chained reasoning, a stronger model can deliver substantially better results.
Hard coding work
Large models earn their keep on complex requirements, unfamiliar codebases, architectural decisions and difficult debugging sessions.
Ambiguous natural language
Some requests do not fit predefined categories. A stronger model is usually better at reading nuance and intent.
Synthesis across many documents
When the answer requires combining information from many sources, model capability matters more.
Agentic workflows
An agent typically has to:
- understand a goal
- plan actions
- choose tools
- inspect results
- recover from failures
- revise the plan
- finish the task
That is far harder than classification, and a stronger model can be entirely justified. The test in every case is the same: use large models where their extra capability produces value you can measure.
The operational cost of a clever architecture
There is a cost that benchmarks never show: architectural complexity. The simplest possible design is one application, one large model, one response:
Application
↓
Large Model
↓
Response
Now imagine optimizing everything at once:
Application
↓
Router
↓
Classifier
↓
Small Model
↓
Confidence Evaluator
↓
RAG
↓
Reranker
↓
Large Model
↓
Validator
↓
Fallback Model
↓
Human Review
This pipeline might well produce a better system, but it also introduces many more components, and every component brings:
- additional monitoring and logs
- new failure modes
- a larger test surface
- extra infrastructure to run
- harder deployments
- more knowledge the team must hold to operate it
Optimization should therefore be deliberate. Building a seven-model architecture to save a few cents per request is rarely a good trade; add each layer only when measurements show it pays for its maintenance.
Evaluate on your own workload, not on leaderboards
Public benchmarks are a starting point, not a decision. Your application has its own benchmark, and it is the only one that counts. For an AI code-review assistant, an evaluation set might include:
- SQL injection vulnerabilities
- race conditions
- null-reference bugs
- authorization flaws
- incorrect exception handling
- performance problems
- architectural violations
For a customer-support assistant, you would measure different things:
- policy compliance
- factual accuracy
- tone
- escalation accuracy
- refusal behaviour
- validity of structured output
The evaluation set should resemble production traffic as closely as possible.
Build a comparison harness
The basic procedure is straightforward: collect production-like prompts with expected outcomes, run each candidate model against them, and compare.
Production-like prompts
↓
Expected outcomes
↓
Run Model A
↓
Run Model B
↓
Compare
↓
Measure
Useful metrics to record for each run:
- correctness and task completion
- how often the model hallucinates
- latency
- tokens consumed and resulting cost
- the share of structured outputs that validate
- refusals and errors
With that in place, choosing a model becomes an engineering decision instead of guesswork, and you can rerun the same harness whenever a provider ships a new model version.
A four-step model selection process
For a new AI feature, a simple, repeatable process works well.
Step 1: Define the task precisely
Do not begin with "which model should we use?" Begin with "what exactly must the model accomplish?" and write the answer down in concrete terms:
Input:
Customer email
Output:
Intent + urgency + recommended workflow
A specification like this, with a clear input and a clear output, is far more useful than a vague goal.
Step 2: Define what "good enough" means
Set explicit acceptance criteria before testing anything. For example:
Intent accuracy >= target threshold
Structured output must always validate
Response should normally arrive within target latency
The actual thresholds depend on the application; what matters is that they exist before you compare models, so the results cannot be rationalized afterwards.
Step 3: Try the smallest viable model first
This is the step teams skip most often. Start small, measure, and stop if it passes. Only move up if it fails:
Small Model
↓
Evaluation
↓
Pass? ── Yes → Ship
|
No
↓
Larger Model
↓
Evaluation
↓
Pass? ── Yes → Ship
|
No
↓
Stronger architecture/model
In effect you are climbing a capability ladder one rung at a time, and each rung has to be earned by a failed evaluation.
Step 4: Optimize the surrounding system
Before moving up another rung, check the rest of the system:
- Does retrieval return the right material?
- Is the prompt unambiguous?
- Is every piece of context actually relevant?
- Is the output validated before use?
- Could plain code take over part of the job?
- Would a cache absorb repeated requests?
- Are there tokens you can trim?
- Could you escalate only the hard requests?
Improvements here sometimes remove the need for a larger model altogether.
Other levers worth pulling before an upgrade
Prompts as contracts
Prompt engineering is not magic, but a poorly specified task can make even a capable model behave badly. Compare a vague instruction:
Analyze this customer message.
with one that defines the expected output and the rules for uncertainty:
Analyze the customer message.
Return JSON with:
- intent
- urgency
- sentiment
- recommended_action
Do not invent information that isn't present.
If the intent is unclear, return "unknown".
The second version gives the model a clear contract: named fields, an explicit prohibition on invented facts, and a defined fallback value. Adding a few examples often improves results further. There is a ceiling, though. No amount of prompt wording gives a model abilities it fundamentally lacks, and when the capability is missing, prompt tweaks deliver diminishing returns. Treat prompting as one layer of the optimization stack, not the entire solution.
Fine-tuning, RAG or validation?
Another common reaction to weak results is "let's fine-tune." Sometimes that is exactly right, but diagnose the problem first:
- If the model lacks current or company-specific knowledge, such as today's policy, retrieval is usually a better fit than fine-tuning, because knowledge changes and retraining is slow.
- If the model does not reliably produce the output format you need, structured prompting plus validation is often enough.
- If the model consistently lacks a specific behaviour across many examples, such as a house style or a domain-specific judgement, fine-tuning becomes attractive.
Matching the remedy to the actual failure saves both time and money.
Caching: the boring optimization that works
Caching is unglamorous and very effective. If users keep asking "what is your return policy?", there is no reason to call a model every time. When the answer is stable, cache it. The C# example below checks a cache first, calls the model only on a miss, and stores the result for 30 minutes:
public async Task<string> GetAnswerAsync(string question)
{
var key = CreateCacheKey(question);
var cached = await cache.GetStringAsync(key);
if (cached is not null)
return cached;
var answer = await model.GenerateAsync(question);
await cache.SetStringAsync(
key,
answer,
TimeSpan.FromMinutes(30));
return answer;
}
Real semantic caching is more sophisticated than exact string matching, since two differently worded questions can deserve the same answer, and you need a policy for invalidating answers when the underlying facts change. The architectural idea still holds:
The fastest AI request is the one you never make.
The same principle applies to deduplication, precomputation, deterministic responses, reuse of retrieval results, prompt-prefix caching where a provider supports it, and plain response reuse. Before paying for more compute, eliminate compute you do not need.
Treat tokens as a resource budget
Once you view an AI system as a distributed system, tokens become one more resource to manage. Traditional services manage CPU, memory, network and storage. AI applications add input tokens, output tokens, context size and inference time, which makes prompt design a performance concern.
Consider sending the full conversation history with every request. The prompt keeps growing, and then retrieved documents, tool outputs, system instructions and previous agent actions pile on top. A simple question can end up carrying an enormous payload. A leaner design selects only relevant history before retrieval and builds a compact context:
Conversation
↓
Relevant history selection
↓
Retrieval
↓
Compact context
↓
Model
rather than forwarding everything the system has ever seen:
Everything we've ever seen
↓
Model
More context is never free: you pay for it in money, latency and, often, answer quality.
Agents multiply every one of these decisions
Agentic systems make model selection even more consequential. A single user request can fan out into many steps:
User Request
↓
Planning
↓
Tool Selection
↓
Search
↓
Database Query
↓
Code Execution
↓
Analysis
↓
Final Response
If every step runs on the most expensive model, costs can explode. Yet the steps rarely need the same capability. A mixed assignment might look like this:
Intent classification → Small model
Simple tool selection → Small model
Complex planning → Large model
Data extraction → Small model
Final explanation → Medium model
This is a heterogeneous AI architecture, and it is where many production systems are heading: not one giant model doing everything, but several models, tools, deterministic components, retrieval pipelines and validators working together. Our breakdown of why agentic AI costs explode explores the cost side of this in more detail.
Think of models as members of a team
A helpful analogy: imagine three engineers. One is very senior, expensive and overloaded. One is experienced and efficient. One is junior but very fast at repetitive work. You would not hand every task to the senior engineer. You would assign work by difficulty:
Simple repetitive task
→ Engineer C
Normal feature
→ Engineer B
Complex architecture problem
→ Engineer A
Models can be treated the same way. A smaller model is not "bad"; it may simply be suited to a narrower responsibility. The core skill is breaking work into pieces. Rather than searching for one model that handles everything, split the workflow so that every component gets the part it handles best. That way of thinking scales much better.
Putting it together: a reference architecture
Combining these ideas, an enterprise AI assistant could be structured like this:
User
|
v
API / Gateway
|
v
Request Router
|
+-------------+-------------+
| |
v v
Simple Request Complex Request
| |
v v
Small Model Planner
|
+------------+------------+
| | |
v v v
Search Database Tools
| | |
+------------+-------------+
|
v
Context
|
v
Strong Model
|
v
Validator
|
+------+------+
| |
Valid Invalid
| |
v v
Response Retry/Fallback
Notice what is not happening: the largest model is not asked to do everything. Simple requests go to a small model. Complex ones pass through a planner that gathers evidence from search, databases and tools, and only then does a strong model work on a curated context. A validator checks the output, and invalid results trigger a retry or fallback. The design relies on:
- a router that picks the path
- retrieval and tools for evidence
- deterministic systems for exact work
- models specialized per role
- validation plus retry and fallback strategies
Here the model is one part of the system rather than the whole system.
Ten mistakes that keep recurring
- Picking a model first and describing the problem later. That order is reversed; pin down the workload before anything else.
- Optimizing for benchmark scores. A public benchmark knows nothing about your business requirements; your own evaluation set matters more.
- Sending every request to the largest model. This adds needless cost and latency; route when workloads vary.
- Solving missing knowledge with a bigger model. If the information is not in the model's training data, retrieval is usually the better fix.
- Asking the model to do deterministic work. Calculations, data storage, authorization checks and business rules belong in ordinary code, not in a language model.
- Assuming a larger context is always better. Extra information can become noise; retrieve what is relevant instead.
- Ignoring latency until production. Measure it from the first prototype.
- Ignoring token usage. A prompt that looks harmless in development can become expensive at scale.
- Expecting a model upgrade to fix an architecture problem. Often retrieval, prompts, tools, validation or the workflow itself is the real bottleneck.
- Never measuring after launch. Models, prompts, user behaviour and data all change, so AI systems need continuous evaluation and production monitoring.
A decision checklist for choosing a model
When a model decision comes up, work through these questions in order:
- Can deterministic code solve it? If yes, write code. Do not use AI simply because it is available.
- Is the task simple and repetitive? Try a small model.
- Does it need private or current information? Consider RAG or tool access.
- Does it require sophisticated reasoning? Evaluate a stronger model.
- Is latency critical? Prefer lower-latency options.
- Is request volume high? Cost and throughput become dominant factors.
- Can hard requests be escalated? If so, consider a model router.
- How expensive is failure? For high-risk tasks, combine stronger models with validation and human oversight.
This checklist is far more useful than asking which model is the biggest one available.
A question worth asking senior AI engineers
A revealing interview question for an AI architecture role is: "Why not simply pick the most capable model on the market for everything?"
A weak answer stops at "smaller models are cheaper." That is true but incomplete. A strong answer brings in capability for the specific task, reliability, latency and throughput, cost, how much context is needed, routing between models, retrieval, work that code should do, evaluation, what failure costs, and the operational burden.
The natural follow-up is: "How would you prove the model you chose is good enough?" The answer should be about evaluation, not opinions, social media threads, leaderboard screenshots or vendor slides. What settles it is your workload, your data and your metrics.
Optimizing an existing application step by step
When an AI feature is too slow or too expensive, resist the urge to swap models immediately. Investigate systematically instead:
- Measure. Collect request counts, input and output tokens, latency, error rate, task success rate and model cost.
- Find the expensive workloads. Identify which request types consume the most resources.
- Remove unnecessary calls. Use caching, deterministic logic, deduplication and precomputation.
- Shrink the context. Drop irrelevant history and documents.
- Improve retrieval. Return better evidence rather than more evidence.
- Try a smaller model. Check whether quality stays acceptable.
- Introduce routing. Send only the difficult cases to stronger models.
- Validate what matters. Apply schemas, rules, tools and human review where appropriate.
- Re-evaluate. Never assume an optimization worked; measure it.
If this looks familiar, it should. It is essentially the same discipline used to optimize conventional software.
The best AI architecture is usually hybrid
The picture of an AI application as a frontend calling an LLM API and returning the response is fading:
Frontend
↓
LLM API
↓
Response
Real applications increasingly look more like a gateway coordinating rules, retrieval, models, tools and validation:
Application
|
v
AI Gateway
|
+-----------+-----------+
| | |
v v v
Rules Retrieval Models
| | |
+-----------+-----------+
|
v
Tools
|
v
Validation
|
v
Application
Such a design blends classic software engineering with machine learning, language models and retrieval, alongside databases, APIs, security, observability and business logic written as code. That is good news for software engineers: AI engineering is not replacing software engineering, it is adding a powerful new component to it.
A related shift in perspective helps: stop asking which model is smartest and start asking which system is smartest. A brilliant model inside a poorly designed system can produce terrible results, while a moderately capable model inside a well-designed architecture can power an excellent product. Good systems make up for a model's weaknesses with retrieval and tools, routing and caching, structured outputs and validation, code for exact work, and ongoing evaluation. In that sense, AI capability is a property of the architecture, not just of the model.
Start small and escalate with evidence
A sensible default strategy for any new AI feature is captured in this flow:
Start
|
v
Define the workload
|
v
Can code solve the problem?
/ \
Yes No
| |
Use code v
Try small model
|
v
Evaluate
|
+----------+----------+
| |
Pass Fail
| |
Ship v
Improve architecture
|
v
Evaluate
|
v
Try stronger model
|
v
Evaluate
It prevents one very common mistake: spending money on a problem that better engineering could have solved.
Sometimes the right model is no model
The lesson is not that small models are better; that would be just as wrong as the opposite claim. The right model is whichever one satisfies the task's requirements while balancing capability and reliability against latency, cost and complexity. Sometimes that is a small model, sometimes a large one, sometimes a combination, and sometimes not an AI model at all. If a request type can be handled with a simple branch like this one, there is no reason to invoke an LLM:
if (request.Type == "PasswordReset")
{
return StartPasswordResetWorkflow();
}
That is not anti-AI; it is good engineering. You would not buy a server with unlimited CPU for a service that needs two cores, or provision a terabyte of memory for a process that uses 4 GB. The same reasoning applies to models: capability you do not need is cost you do not need.
Key takeaways
- Replace "what is the biggest model we can afford?" with "what is the simplest architecture that reliably solves this problem?" That question naturally leads you to routing, retrieval, caching, deterministic code, evaluation, latency and failure handling.
- Judge models by cost per successful task, weighed against the consequence of a wrong answer, not by price per call or parameter count.
- Treat missing knowledge as a retrieval problem and arithmetic or rules as a software problem; neither is fixed by a bigger model.
- Default to the smallest model that passes an evaluation built from your own production-like data, and escalate only with evidence.
- Keep the architecture as simple as the savings justify, and keep measuring after launch because models, prompts and users all change.
Design around the problem first, and pick a model to suit that design afterwards. Sometimes that model will be huge, sometimes surprisingly small, and sometimes the smartest decision is not to call a model at all.