Choosing an LLM Provider: Constraints, Routing and Reversible Design

Teams keep asking which model is the best one, as if choosing were a taste test. That question has a short shelf life. Provider prices changed twice in the first half of 2026, and every major lab shipped new model versions in the same stretch. A company that locked itself to one vendor in January has already seen that bet go stale.

A better question has three parts. What does this workload demand? Which class of model meets those demands? And how cheaply could we switch in six months? If moving to a different provider means a rewrite, you haven't made an architectural choice. You've picked up a dependency.

What follows covers the constraints that really settle the matter, plus a code layout that keeps the choice reversible.

Five constraints that settle it

Go through these before opening any benchmark table.

Where your data may travel. Closed API models send your data through the vendor's infrastructure with each request. For regulated sectors, patient records or public-sector contracts, that fact alone can end the discussion. Self-hosting or in-region deployment then stops being a cost matter and turns into compliance.

Cost at production volume. Demo traffic hides the spread. Budget models sit around $0.10 per million input tokens, while frontier models reach $25 to $30 per million output tokens. At about a million conversations a month, hosted models land somewhere between $15,000 and $75,000 monthly. A small fine-tuned model run on your own hardware can handle that load for a few hundred dollars, provided the task is narrow and repeats.

Real task difficulty. Most business workloads are sorting, extracting, summarizing and routing. None needs frontier-grade reasoning, so keep the pricey tier for the hard minority.

Latency tolerance. A person waiting for a chat reply and an overnight batch have nothing in common. Paying for speed you don't need is the most frequent silent overspend.

Appetite for operations. Open-weight models cost nothing to download, but running them means renting GPUs, from about $0.50 up to $5.00 per hour as models get bigger, and paying engineers to deploy, patch and monitor the stack.

Step 1: build the seam first

Define the abstraction before choosing anyone. All other code should talk to your interface and never to a vendor SDK.

class ModelProvider(Protocol):
    def complete(self, messages, *, max_tokens, tools=None) -> Completion: ...

class Router:
    def __init__(self, providers: dict[str, ModelProvider]):
        self.providers = providers
    def complete(self, messages, *, tier="standard", **kw):
        name = MODEL_FOR_TIER[tier]                # config, not code
        try:
            return self.providers[name].complete(messages, **kw)
        except (RateLimited, ProviderDown):
            fallback = FALLBACK_FOR_TIER[tier]     # different vendor entirely
            return self.providers[fallback].complete(messages, **kw)

This gives you two things. A provider swap becomes a config edit and not a project. And one vendor's outage no longer takes down your product, something teams underestimate until the day it happens.

Keep vendor oddities inside each adapter. As soon as a provider's parameter names creep into business logic, the seam is gone.

Step 2: route by job, not by favorite

Sending every request to one model is the costliest habit in production AI. Split the work into tiers.

MODEL_FOR_TIER = {
    "bulk":     "small_local",      # classification, extraction, tagging
    "standard": "mid_hosted",       # summaries, drafting, most chat
    "hard":     "frontier_hosted",  # multi-step reasoning, ambiguous cases
    "sensitive":"self_hosted",      # data that cannot leave your estate
}

def choose_tier(task, payload):
    if contains_regulated_data(payload):
        return "sensitive"                      # compliance overrides cost
    if task in {"classify", "extract", "tag", "route"}:
        return "bulk"
    if task == "reason" or payload.get("complexity") == "high":
        return "hard"
    return "standard"

The compliance check goes first on purpose. Cost and quality can be traded against each other. Where data may be processed can't.

Step 3: measure on your own workload

Public leaderboards show how models do on public leaderboards. They correlate only loosely with how a model copes with your documents, your jargon and your languages.

Run your own bake-off using the same golden set that guards against regressions.

def bake_off(candidates, cases):
    results = {}
    for name, provider in candidates.items():
        passed, cost, latencies = 0, 0.0, []

for case in cases:
            t0 = time.monotonic()
            out = provider.complete(case["messages"], max_tokens=800)
            latencies.append((time.monotonic() - t0) * 1000)
            passed += grade(out.text, case["expected"])
            cost += price(name, out.usage)
        results[name] = {
            "accuracy": passed / len(cases),
            "cost_per_1k_calls": (cost / len(cases)) * 1000,
            "p95_latency_ms": percentile(latencies, 95),
        }
    return results

Judge three things at once: accuracy, price and speed. A model two points more accurate at four times the cost isn't clearly better. One slightly cheaper but three times slower fails a customer-facing feature whatever its score says.

Grade instruction following and tool-calling reliability too. In most production systems, calling the correct function with valid arguments counts for more than raw intelligence.

Step 4: price it before you commit

def monthly_estimate(name, calls_per_month, avg_in, avg_out, cache_hit_rate=0.0):
    p = PRICING[name]                              # per million tokens
    billable_in = avg_in * (1 - cache_hit_rate) + avg_in * cache_hit_rate * p["cache_ratio"]

return calls_per_month * (
        (billable_in / 1_000_000) * p["input"]
        + (avg_out / 1_000_000) * p["output"]
    )
# Always run it at 10x expected volume before signing anything.

Two levers shift the result a lot and are often forgotten. Batch processing usually cuts the rate in half for work that isn't urgent. Prompt caching can reduce the cost of repeated input by as much as roughly 90 percent, which is huge when every call carries the same long context.

Three families of models

Closed API models get you to production fastest and lead at the frontier, with no servers to run. In exchange, your data passes through their systems, your bill scales linearly without end, and their pricing and retirement decisions are yours to absorb.

Open-weight models through a hosting provider cost less, travel better and improve quickly. The median launch price of flagship open models climbed from about $0.48 in 2024 to $1.35 in 2026, still far below closed tiers. One caveat: outside hosts often serve models at lower numerical precision to save money, which can trim quality a little, and their reliability promises differ from the original lab's.

Self-hosting gives full control of your data and no per-token charge. You pay with GPU capacity and engineering time, and you're responsible for patching, scaling and uptime. It's justified by sovereignty rules, or by narrow high-volume tasks where a small tuned model beats a general one on price and accuracy.

Most mature systems use all three, and that isn't indecision. It's the design.

Security points

• Treat residency as an input to the design. Decide which kinds of data may be processed where before you shop for a provider. Adding this late is how projects die in compliance review.
• Read the data handling terms closely: retention periods, whether inputs train future models, the subprocessor list and which region actually answers your requests. Enterprise contracts usually differ from defaults, and defaults are what you get if nobody asks.
• A prompt is an outbound data flow. Anything inside has left your estate, so redact before sending and not after.
• Self-hosting relocates risk without erasing it. Now the weights are yours to protect, every inference endpoint is exposed to attack, and updates become your job. An unauthenticated internal inference endpoint is a common, serious finding.
• Handle API keys like production credentials: one per environment and service, rotated, narrowly scoped and capped by spend limits. A leaked key means a data risk plus an unlimited invoice.
• Pin model versions. Providers retire and upgrade models quietly, and a model that shifts under you can break behavior you verified months ago. Fix the version and rerun the golden set before accepting any change.

Checklist

• One provider-neutral interface, with vendor details confined to adapters
• Model choice kept in config, changeable without a deploy
• A fallback to another vendor for outages and rate limits
• Work routed by tier, with the compliance question before the cost question
• A golden set run on each candidate for accuracy, cost and p95 latency
• Tool calling and instruction following graded along with answer quality
• Costs modeled at ten times the expected volume
• Batching and caching wherever the workload permits
• Residency decided per data class before a provider is chosen
• Versions pinned and revalidated before upgrades
• Keys scoped per service with spend caps

Closing thought

Whoever says one model is the answer is describing this month, not your system.

Strong teams treat model choice as a routing decision inside a stable architecture. Cheap work goes to cheap models, regulated data stays where it's allowed, evaluation uses their own tasks and not someone's leaderboard, and any of it can change in an afternoon.

Struggling teams crowned a favorite, threaded it through every layer of the product, and end up rewriting whenever the market shifts. With prices having reset twice within six months, that adds up to a heavy tax.

Choose per workload, build the seam, and redo the numbers every quarter.