Swapping LLM Models Safely: Human Labels, Per-Step Metrics, Effort Settings

Changing a model name in a config file looks like a five-minute job. In a multi-step LLM pipeline it rarely is, and the reason is usually not the new model: it is the measurement you use to judge it. Using a realistic migration of a filtering pipeline off an older model family, you will see how to build trustworthy ground truth, attribute losses to individual steps, tell a stale prompt from a weaker model, and treat reasoning effort as a per-task setting rather than a global dial.

The concrete trigger in this scenario is a deprecation. OpenAI has announced 11 December 2026 as the retirement date for four models: gpt-5, its mini and nano variants, and gpt-5-pro. That is the schedule at the time of writing, so confirm it on the official deprecations page. The pipeline in question relied on the two smallest of those, so moving was mandatory. Two replacement options were on the table: gpt-5.4 in its various sizes, and gpt-5.6-luna. Model names, defaults and pricing programmes change quickly, so treat the specifics below as a snapshot and the method as the durable part.

The pipeline being migrated

The system ingests roughly 2,600 articles per month and narrows them to about 120 candidates, which a person then reviews manually. That reduction is not a single prompt. It is an agent built with LangGraph, around a dozen nodes in the graph, ten of which are distinct model calls. Three of those calls carry most of the story: a relevance gate at the entrance, then two classifiers, one for content type and one for topic. Behind them sit a novelty check, a scoring step and a summarisation step.

One property of the design shapes every decision that follows. The first node exists to remove noise rather than to judge quality, and its two error types have very different costs:

• A false keep is cheap. A stronger model downstream, or the human reviewer at the end, will catch it.
• A false drop is permanent. The article is marked as seen and never re-enters the pipeline.

Any evaluation that treats those two errors symmetrically will mislead you about the front of the pipeline.

The configuration that shipped

The end state was lopsided on purpose. Nine of the ten calls moved to gpt-5.6-luna, which has behaved well so far. The relevance gate stayed on gpt-5.4-nano; justifying that exception consumed most of the investigation. And every call now sets its reasoning effort explicitly instead of inheriting the model default, and that choice ended up mattering more than the model selection itself.

Cost narrowed the shortlist before quality was even assessed, but not via list prices. Nearly all of the traffic is billed under OpenAI's data-sharing programme, and luna draws from the same, larger daily allowance as the cheapest models already in use. The monthly bill therefore stays at a few cents either way. The stronger models of the same generation belong to a much smaller allowance that this workload would burn through by midday, which ruled them out regardless of quality.

What follows are four surprises, in the order they surfaced. That order also happens to be a sensible checklist for your own migration.

Lesson 1: do not use the old model as the baseline

The first evaluation compared each candidate configuration against the retired model's output: how much of the previous result does the new setup reproduce? That comparison reported a 31-article regression and triggered a search for a model defect that was never there.

In hindsight the problem is obvious. The pipeline's first pass generates candidates, and between 40 and 60 percent of them are later removed by the human reviewer. Measuring agreement with that output means measuring agreement with a filter already known to be mediocre. A candidate is rewarded for copying the old model's mistakes and penalised for fixing them.

Human decisions are already recorded ground truth

A far better label set was sitting in version control. The human review step deletes entries from the candidate list, and each deletion lands as a commit. In the month used for testing, the first review pass shrank 122 candidates to 72, and later passes removed more. Diffing that commit against its parent yields two labelled groups: articles a person kept and articles a person rejected.

Rescoring the 31 "lost" articles against those labels changed the picture completely. Forty-three percent of them were articles the reviewer had thrown away anyway. Measured against human judgement, the actual loss was 15 good articles, roughly half of the apparent regression. Even better, because the labels can be joined to per-step decisions, they show which node dropped each of those 15. (These labelled figures come from the setup with gpt-5.4-mini, not from the luna setup that shipped, because the luna run was stopped early.) A single node accounted for the movement; no other step changed by more than one article. A localised, 15-article problem in one prompt is far easier to own than an unexplained 31-article drop.

The general lesson: whenever a person makes the final call in your system, their call is the ground truth, and there is usually a record of it already. Look at approval queues, moderation overrides, reclassified support tickets, or edits people make to generated drafts. The existing article on evaluating RAG systems by failure stage makes a similar case for attributing errors to the component that caused them.

Per-step measurement also exposes bugs unrelated to the migration. In this case it revealed that the novelty check invents topic names that do not appear anywhere in the list it receives, a different fabricated name for each of its 24 rejections, and it does so with both the old and new models.

Lesson 2: a stricter model may be reading your prompt correctly

Once the loss was pinned on the topic filter, the tempting move was to adjust its prompt until the new model behaved. Here that would have done real damage, because a stronger model further down the graph loads those very prompt files. Relaxing a rule so that a first-stage model passes more articles quietly weakens the stage that does the serious judging, and nothing measured at the first stage would ever show it.

So instead of editing the prompt, the investigation asked the model to justify itself. For every article the topic filter rejected, gpt-5.4-mini's stated reason was captured, about twenty calls in total.

The explanations were coherent. They named specific rules and cited the prompt faithfully. In one case the model repeated, almost word for word, a rule that ruled out AI cost optimisation "even when implemented at an API gateway", and used it to drop an article the reviewer had actually kept.

The old gpt-5-mini had simply never enforced that rule. On inspection, two of the rules it had been ignoring really were outdated. One lacked an exception for messaging patterns built inside an application; the other was written before AI gateways had become infrastructure worth covering. Those two were corrected, the others left untouched, and the fix went into its own commit, separate from the model change, so each could be measured independently.

A cheap test that separates two opposite diagnoses

This is the check worth running first in any migration. When a newer model rejects more items at a judgement step, the cause is often better instruction-following colliding with a prompt that has drifted for a year. Read a sample of its stated reasons:

• When the explanations quote your real rules correctly, the prompt has gone stale, and correcting it helps every stage that shares it.
• When the model invokes rules that clearly do not match the item, your prompt is sound and the model is at fault.

Those conclusions point in opposite directions, and a raw rejection count cannot distinguish between them.

Lesson 3: a stricter verdict is not automatically a worse one

The relevance gate told a similar story from a different angle. At that node luna turned away double the number of articles that gpt-5.4-nano did. The obvious reading was that luna is simply worse at this task, and that almost became the decision.

The gate, however, is really answering three separate questions: is it recent enough, is the language English, and does the subject fall within scope? Only the combined verdict had been examined.

Splitting the verdict apart was revealing. For the language question the models matched perfectly, flagging the identical four articles. The whole gap lived in the subject question, where luna rejected 64 articles and gpt-5.4-nano rejected 32.

Reading luna's reasons showed coherent, mostly defensible judgements: among its rejections were a library for Go worker pools, a platform for building voice agents, and a write-up on migrating databases in the cloud. Nothing was broken. Luna interprets "primary focus" more narrowly than the prompt's author intended.

That calls for a different remedy. A broken check should be repaired. A defensible but stricter reading of a vague prompt, placed at the one gate that feeds everything downstream, where false drops are permanent, is better routed around. That is why this single call stayed on gpt-5.4-nano. More generally, an aggregate score only tells you that something changed. If a step answers several questions, log each answer separately before forming any theory about the model.

Lesson 4: reasoning effort does not transfer between models or tasks

Once gpt-5.4-nano was chosen for the gate, the remaining question was how much it should think. Reasoning models expose an effort parameter that roughly controls how much internal deliberation happens before the answer. The working assumption was that effort behaves like a quality dial that always turns the same way, with each model responding in its own characteristic way.

The measurements showed two quite different patterns:

• At the relevance gate, turning effort up almost doubled the rejections from gpt-5.4-nano, while luna did not react to the setting at all.
• At the topic filter, the same luna model moved by 49 articles across the effort range and was still dropping at the highest setting.
• The retired gpt-5-nano was the most permissive configuration of all.

So a statement like "luna needs high effort" is not a reusable fact. It held for luna on one task and meant nothing for luna on another. Effort is a property of a specific model on a specific task, and every such pairing needs its own measurement.

A bigger model does not substitute for that measurement. A separate 80-article probe of the relevance gate, with every model at medium effort, produced 31 rejections from gpt-5.4-mini, 34 from luna and 30 from gpt-5.4-nano. A larger model barely moved the result; switching gpt-5.4-nano's reasoning off entirely moved it a lot.

Effort also has a cost profile unlike anything else in the bill, because it can grow without a hard ceiling. Raising the topic filter by one effort level saved 10 rejections at roughly three times the token usage, a trade that was declined.

What was not measured

Being explicit about gaps is part of the method.

• The shipped luna configuration was never scored against the human labels; that run was stopped early. It has run in production without a single failed model call, which says nothing about quality loss.
• A single data point hints that luna does worse than gpt-5.4-mini on content-type classification, dropping 9 good articles where gpt-5.4-mini dropped 5, at an effort level that was not recorded. If output volume thins, that is the first node to inspect.
• The topic-filter figures were produced by a harness that sends the full article set directly to that node. In the real pipeline, earlier steps have already removed most inputs, so the absolute numbers are inflated. The relevance gate is immune to this, since it receives the full stream in production as well. The calibration control for the topic filter was never run, so the shape of its curve is reliable but its absolute values are not.
• Noise cuts both ways. An earlier conclusion that one model was "genuinely better", based on a 10-point gap, had to be withdrawn when a third run of the identical configuration landed on the other side. The fix is procedural: quantify the drift between two runs of the same setup before believing any difference between two setups. Only the 15-article loss exceeds that threshold, and it is confined to one node.

The measurement details

Two sample sets were used. Effort experiments sent all 198 cached articles directly into one node, so nothing upstream could vary. End-to-end experiments pushed a 200-row sample through every node; it included 72 human-approved articles and 49 human-rejected ones.

Scored against those human labels:

• Retired setup (gpt-5-mini plus gpt-5-nano): 48 good articles retained out of 72; 27 of the 49 bad ones let through.
• gpt-5.4-mini handling the judgement nodes, with the prompt fixes applied: 33 of 72 retained; 18 of 49 let through.
• Shipped configuration (luna on nine of ten calls): not yet scored against either set.

That last entry stays empty until the monthly review happens, because labels only exist once the prune commit lands, and September had not been pruned at the time of the analysis.

Early production signal

Production data offers a partial substitute. Luna went live during 30 August. The inclusion rates for comparable windows look like this:

• Previous models across 1–29 August: 4.55 percent (121 included out of 2,658).
• Previous models in the final eight days before switching: 4.86 percent (32 out of 658).
• Luna from 30 August through 6 September: 4.08 percent (26 out of 638).

Comparing the two matched eight-day windows with a two-proportion z-test yields 0.69, which is well inside noise. For perspective, the previous setup ran at 3.98 percent during the opening eight days of August, a bigger swing within its own history than the change luna introduced. A week and a day is a short window, and inclusion rate does not measure quality, but the most feared outcome, luna silently starving the pipeline, has not appeared.

Luna's time in production also produced the first per-node breakdown of rejections. Luna excluded 603 articles in that window. The relevance gate was responsible for 42 percent of them, the content-type filter for 38 percent, the topic filter for 16 percent, and the novelty check for 3 percent. By contrast, the previous setup excluded 2,526 articles over 1–29 August without storing a single reason, so reconstructing August meant replaying articles rather than simply querying production data. If your pipeline does not log a per-step rejection reason today, adding it is the cheapest improvement you can make before a migration.

Limits of the ground truth and of a single run

The human labels have two built-in limits. They only cover articles the old pipeline let through, so a new configuration can never get credit for rescuing something the old one wrongly dropped. And roughly 79 rows in the 200-row sample have no human verdict whatsoever.

Run-to-run variance is large. Three repeated runs of one unchanged setup over the identical 200 rows produced rejection rates of 37, 48 and 55 percent at a single node, a spread far wider than sampling noise would explain. Repeating a full end-to-end run of one setup produced four new items and lost four others. Replaying the retired setup against what it had itself produced in production recovered just 75 of its 121 labelled articles, so it agreed with its own history only about 62 percent of the time. In practice, one run can only distinguish differences of around 20 points at a node, or around eight items across the whole pipeline.

A better approach for validating individual changes: check each change on precisely the items it is meant to fix, together with a small frozen set it should leave alone, and rely on the broad run only as a coarse alarm for large collateral damage. One prompt fix illustrated why. It pushed four target articles past the node it repaired, yet barely changed the final output, because three of those four were rejected later for unrelated reasons.

Where the tokens actually go

Reasoning spends output tokens, but output is not what consumes the allowance. Output made up 13 percent of all tokens with the previous models and less than 3 percent with the new ones; the prompts are long, so input dominates.

At the time of writing, the data-sharing programme offers two daily pools. The larger one, 2.5M tokens a day, applies to luna and to both the nano and mini sizes of gpt-5.4. The smaller one, 250k tokens a day, applies to the mid-size and larger models. On a normal weekday the pipeline consumed roughly 1.95M tokens with the previous models and roughly 1.15M with the new lineup. The exported month, mostly old configuration plus eight days of the new one, would have cost about $4 at flex rates without any allowance.

Per thousand requests, the models differ widely:

• gpt-5-nano: $0.32
• gpt-5.4-nano: $0.52
• gpt-5.6-luna: $0.65
• gpt-5-mini: $1.49
• gpt-5.4-mini: $3.29 (small sample)

Luna does not win on price per request, and that is acceptable. It costs far less per request than either mini model and draws from the same pool as the nano models.

The prompt-cache write surcharge

The price list hides one detail worth reading carefully. Among the models considered, only luna bills for cache writes, priced 25 percent higher than its normal uncached input. Since requests here trickle in over the whole day instead of clustering, cache entries expire between requests: 57 percent of luna's input was billed as cache writes and only 41 percent as cache reads. Caching still pays, reducing the input bill by 23 percent compared with no caching, but that is a long way from the 90 percent the read discount implies. If your workload is bursty, you will see more of the discount; if it trickles, budget for the write premium.

API traps that fail every request

A few parameter behaviours turn a rename into an outage. As reported at the time of writing (verify against current documentation):

• Default reasoning effort is not consistent across generations: medium for gpt-5, none for the 5.4 family, and back to medium for 5.6. A bare model rename therefore silently changes how much every call reasons.
• The top effort level is available only on 5.6 models; asking for it on a 5.4 model makes every request fail.
• On the chat completions endpoint, any effort above none combined with function tools is rejected; such calls must move to the responses endpoint. Since luna's default is medium, tool calling with luna on chat completions is impossible unless effort is lowered explicitly.

None of these can be recovered at runtime. Typical retry logic handles server errors, timeouts and rate limits, not invalid requests, so a bad model-and-effort pairing fails on every row of a run instead of once. Check the model and effort combination when the process boots, before the first row is handled. For background on the two endpoints, see the existing article on items versus messages in OpenAI's Responses API.

Key takeaways

• Never score a replacement model by its agreement with the model it replaces; use recorded human decisions as ground truth.
• Attribute every loss to a pipeline step, and split multi-question steps into separate, logged verdicts.
• Before retuning a prompt, read the new model's stated reasons. Accurate citations of your rules mean the prompt is stale, not the model.
• A stricter but defensible model at a gate where drops are permanent may be better routed around than fixed.
• Reasoning effort belongs to a model-task pair. Set it explicitly everywhere, measure it per step, and remember it is the one cost without a ceiling.
• Quantify run-to-run drift first; only trust gaps that exceed it.
• The single most useful artefact to request before approving a migration is not a rejection count but a handful of the model's own explanations for what it dropped.