Training Enterprise AI Agents: Choosing Between SFT, DPO, GRPO and RAG

Picture an analyst asking an enterprise agent a question ten days ago: why did sales fall in North America last quarter? The system asked what "last quarter" meant and settled on Q2 FY2026. It mapped North America onto the approved regional hierarchy, produced tenant-bound SQL, tried several hypotheses and delivered a cited report.

Today the analyst asks something similar. The agent still remembers the correct scope, yet it emits malformed tool JSON, spends paragraphs explaining a hypothesis that already failed, and cites a draft finance note with total confidence.

At this point many teams decide the model needs fine-tuning. Sometimes that's right. Quite often they're about to train the wrong thing.

Sort out what went wrong. A region mapping is a fact, a finance note is governed knowledge, and a SQL schema is an interface that keeps shifting, so all three live in memory, retrieval and tool contracts. The weights should hold the steadier habits instead: emitting valid tool calls, following the evidence protocol, recovering after an empty result, favoring conclusions that carry citations, and refusing to answer when the evidence the agent may see falls short.

That split underlies all enterprise adaptation:

Changing facts go to retrieval, durable context to memory, actions to tools, and repeatable behavior to training.

A model update isn't a notebook result. It's a governed release, and its evidence trail begins before the first optimizer step and has to survive a production rollback.

What follows looks at the usual post-training toolbox from the viewpoint of an enterprise agent, a very different target from a benchmark model. The ground covered:

• deciding whether a defect belongs in prompts, retrieval, memory, tools or weights;
• building a data plane that's lawful and resistant to contamination;
• choosing among continued pretraining, supervised fine-tuning, LoRA, QLoRA, DoRA and full fine-tuning;
• matching DPO, IPO, KTO, ORPO and SimPO to the shape of the preference data;
• telling apart the problems that PPO, GRPO, DAPO, GSPO, RLOO and verifiable-reward training each solve;
• using teacher-student distillation to turn frontier capability into deployable economics;
• registering, evaluating, canarying, monitoring and rolling back adapters as production artifacts.

A single running case threads through everything below: INV-22041, a multi-agent sales investigation for the tenant acme-eu. Four logical stages make it up (Enrichment, Data Sufficiency, Hypothesis Testing and Reporting). Every stage has tools and sub-workflows of its own, yet the training contract stays visible at each stage boundary.

When training is the right fix

Reach for training when the desired change recurs, can be measured, and is stable enough to sit in weights, and when enforcing it anywhere else would be pricier. It's usually the wrong first move when the thing changing is a policy, a catalog, a customer's name, today's price, a permission, or a tool schema that a constraint could handle.

Pick the least irreversible layer that closes the defect:

1. A prompt or system pack handles explicit behavior the base model already follows reliably.
2. RAG handles current enterprise knowledge that must be attributable.
3. Memory holds user-stated preferences, durable mappings and earlier approved outcomes.
4. Tool and schema controls give determinism, authorization and control over side effects.
5. SFT or PEFT targets recurring response structure, tool grammar, recovery patterns and domain style.
6. Preference optimization applies when several outputs are valid but one is consistently better.
7. Reinforcement learning applies when success can only be scored after a sequence of decisions.

Run INV-22041 up the same ladder. Knowing that "North America" maps to the approved region node NA_01 is a job for governed memory or metadata. Binding tenant_id, using the read-only semantic layer and returning typed JSON in every SQL request is part runtime contract and part trainable tool behavior. Favoring a short cited explanation over a persuasive uncited story is an excellent target for preference training.

A frequent slip: fine-tuning on the newest enterprise documents. The model can memorize stale or restricted statements while losing provenance, deletion rights and authorization semantics.

Pick the method from the supervision you hold

Teams often begin with an algorithm name. Begin instead with the evidence you really have.

• With prompts paired to gold answers or gold tool traces, use SFT.
• With chosen and rejected responses, reach for DPO-class methods.
• With independent labels marking outputs desirable or undesirable, and no pairing, KTO is a natural fit.
• If a deterministic grader can score every rollout (SQL validity, an exact answer, unit tests, policy checks), verifiable-reward optimization such as GRPO or RLVR is the fit.
• When quality is subjective and can't be reduced to a rule anyone trusts, the classic reward model plus PPO setup stays relevant.

None of this is ideology. Each choice states what your data is able to prove.

A release invariant worth keeping

For every training example, record a tuple holding the example's id and where its sources came from, why consent was given and for what purpose, which tenant it belongs to, the policy version, the hash of the transform applied, the split it landed in, and which labeler or grader version produced it. After deployment, that tuple answers the awkward questions. Where did a given behavior come from? Was the example authorized? Could it leak across tenants? Which transformation produced it? Did it also appear in evaluation?

The enterprise data plane

Most post-training failures start before training does. A polished objective can't rescue examples that are contaminated, unauthorized or poorly specified.

Provenance comes before preprocessing

Every raw source needs an owner, a collection method, a license or consent basis, an allowed purpose, a retention rule, a geography, a sensitivity class and a lineage identifier. Don't discard that metadata when documents or traces become training examples. The finished example has to stay traceable to the original sources and to every transformation applied.

Privacy, secrets and data minimization

Redaction is necessary, and it still isn't enough. A dataset can have no obvious names and yet encode confidential business logic, rare identifiers, customer-specific values or recoverable secrets. Cover these points:

• scan for PII and secrets both before and after transformation;
• keep tenants isolated through sampling, storage and evaluation;
• filter by purpose and not just by access;
• check for near-duplicates against test sets and protected corpora;
• record deletion lineage so a rights request reaches raw data, derived examples, checkpoints, adapters, caches and evaluation artifacts.

Contamination can be measured

A simple operational metric works: the contamination rate is the count of contaminated evaluation items divided by the total number of evaluation items. "Contaminated" covers exact duplicates, near duplicates, paraphrases that carry the answer, and synthetic examples derived from evaluation items. For an evaluation that defines a release, the target is zero.

Human labels need a contract

A preference dataset with no rubric turns into a dataset of annotator taste. For the reporting stage, a useful rubric ranks these in order:

1. authorization and privacy compliance;
2. factual support and correct citations;
3. statistical validity;
4. calibrated uncertainty;
5. completeness;
6. concision and style.

Agreement should be measured. For two raters, Cohen's kappa is the observed agreement minus the agreement expected by chance, divided by one minus the chance agreement. The observed part is the share of labels on which the raters match, and the chance part comes from each rater's label frequencies. A low kappa means the rubric, the examples or the rater training needs repair, and extra epochs won't fix a weak signal.

What a production training record looks like

The useful artifact isn't a bare prompt and answer pair. It's a record that keeps governance joins and evaluation joins possible:

{
  "example_id": "inv22041-sft-01842",
  "tenant_scope": "synthetic-enterprise",
  "purpose": "tool_format_and_recovery",
  "messages": [
    {"role": "user", "content": "Why are sales down in the last quarter?"},
    {"role": "assistant", "content": "I will use the previously approved Q2 FY2026 scope and test the cited hypotheses."}
  ],
  "tool_trace": {
    "name": "query_semantic_layer",
    "arguments": {"tenant_id": "acme-eu", "region_id": "NA_01", "period": "FY2026-Q2"}
  },
  "source_ids": ["memory:m-492", "schema:v17", "policy:p-31"],
  "transform_hash": "sha256:…",
  "split": "train"
}

The values are made up deliberately. The layout isn't: one object carries the content along with who authorized it, where it came from, and how it's identified in evaluation.

SFT, continued pretraining and mid-training

People lump these techniques under "fine-tuning," yet each one optimizes a different data distribution.

Supervised fine-tuning

SFT makes the model imitate a target response or trajectory. The loss is a negative sum over positions: a mask times the log-probability the model assigns to the target token, given the input and the earlier target tokens. In that formula x is the input context, yₜ is the target at position t, the mask mₜ switches on (1) for tokens being supervised and off (0) for the prompt, padding and anything excluded, and θ stands for the trainable parameters.

System, user, tool-result and padding tokens supply context without generating gradients. Only the chosen assistant targets contribute to the supervised loss.

The mask matters. If user and system tokens slip in by accident, the model learns to reproduce the conversation instead of answering it.

What the cross-entropy objective is really doing

At each supervised position the model produces one logit per vocabulary token, and a softmax turns those logits into probabilities. In the softmax, the probability of a token is its exponentiated logit divided by the sum of exponentiated logits across the whole vocabulary, so the results are positive and add up to one.

Cross-entropy then bills the model by the negative log of the probability it gave the correct token. A correct token at probability 0.8 costs about 0.223. At probability 0.01, it costs about 4.605. Confident mistakes therefore produce much bigger gradients than predictions that are already right.

One consequence matters a lot in enterprises. A modest set of high-quality tool examples can firmly repair a systematic formatting defect as long as the target is unambiguous. When two examples show semantically identical requests paired with contradictory tool schemas, the loss demands incompatible predictions.

Sequence loss, token loss and business importance differ

Sequence loss is normally averaged over the supervised tokens, meaning the masked sum of token losses divided by the supervised-token count. Because a 500-token report contributes about fifty times the token losses that a 10-token refusal does, a dataset with equal counts of the two can still end up dominated by reports while optimizing.

When refusal correctness, tenant binding or tool selection carries business weight, example counts don't express that importance by themselves. The options are:

• balance batches by behavior class;
• weight examples or chosen tokens;
• shorten verbose targets;
• oversample rare boundary cases;
• separate generation quality from contract checks in evaluation.

Any weighting must stay explicit. A sampler nobody recorded is part of the objective even though the loss equation never mentions it.

Teacher forcing and exposure bias

Under SFT every token gets predicted after the correct earlier target tokens. At runtime the model instead reads what it generated itself, so one small slip can land it in a state training never showed.

For an agent, that gap shows up as cascading tool failure:

1. The model emits an entity identifier that is slightly wrong.
2. The tool returns an empty result.
3. The model has never seen that state.
4. It invents a result and doesn't repair the query.

Include recovery trajectories along with the perfect demonstrations. On-policy distillation and reinforcement learning tackle the same distribution shift more directly later on.

Masking tool conversations correctly

A realistic transcript holds system policy, user language, assistant reasoning, tool arguments, tool results and a final answer. Decide which tokens should teach behavior:

• system and user tokens normally get mask 0;
• assistant tool calls normally get mask 1;
• raw tool output usually gets mask 0, since it's an observation from the environment and not language the assistant should imitate;
• assistant analysis or hidden reasoning follows the model provider's rules and the deployment policy;
• the final answer gets mask 1.

Supervise tool output by accident and the model may learn to fabricate database results. Exclude tool arguments by accident and report style can improve while tool reliability stays exactly where it was.

Packing without leakage between examples

Say examples A, B and C run 700, 900 and 1,100 tokens. Put each in its own 4,096-token sequence and, on average, only about 22% of allocated positions get used. Packing them raises utilization, though a combined sequence needs guardrails:

• the model's own position handling has to be respected;
• anything that shouldn't be learned (padding, excluded roles) gets the framework's ignore label, usually −100;
• attention shouldn't cross from one example into another if the method treats them as independent;
• debugging and deletion requests still need each example's metadata to be recoverable.

A frequent slip: switching on packing=True and assuming the library default fits the chat template, the attention implementation and the loss policy. Check one batch by eye: decoded inputs, labels, masks, example boundaries and supervised-token counts.

For an enterprise agent, the SFT examples worth most are those with valid tool arguments, clarification behavior, structured refusals, recovery from empty results, citation grammar and hand-offs between agents. Put negative boundary cases into evaluation even when the training data holds only desired responses.

Continued pretraining

Continued pretraining applies the plain next-token objective to domain text. Use it when the base model has never really met a field's vocabulary, notation or general spread of text. Teaching an operational policy exactly is something it does badly.

Catastrophic forgetting is the enterprise risk here: perplexity on the domain gets better while the model slips on following general instructions, handling other languages, staying safe or using tools. Keep replay data around and run broad capability floors.

Mixing domain data with replay data

Call the new domain distribution D_domain and a sample of the capabilities that must survive D_replay. A simple mixture objective adds a weighted expected loss on domain data to a weighted expected loss on replay data. Neither weight is a universal constant. A tightly specialized local model may accept a heavier domain weight, whereas a shared enterprise foundation model usually needs a stronger preservation term.

Watch at least three curves separately: domain validation loss, replay validation loss and downstream behavioral evaluations. A lower combined loss can mask a replay curve that's getting worse, so stop conditions should include preservation floors as well as domain convergence.

Learning rate sets the blast radius

Continued pretraining changes many more parameters than a small adapter does, and a learning rate that suits an adapter can damage a dense base model. Use conservative schedules, checkpoint often and evaluate the intermediate checkpoints, because the final optimizer step isn't automatically the best enterprise checkpoint.

Token count shapes the effective change as well. A tiny learning rate across billions of tokens can shift a model further than a larger one on a small curated set, so log the total tokens and optimizer steps and not just the epoch count.

Mid-training

Mid-training falls between broad pretraining and final post-training. Curated long-context, reasoning, multimodal or agent traces shift capabilities before alignment happens. For INV-22041 that could mean long investigation traces where the agent sees an empty SQL result, wonders if the time grain or semantic mapping is the culprit, fixes the query, updates its hypothesis set and writes a compact evidence ledger.

Packing places several short sequences in one context window and so improves utilization, but examples then need correct attention and loss boundaries. A bad packer lets one customer's example attend to another's without any warning.

SFT or mid-training: a diagnostic

Ask whether the behavior you want can be shown in a fairly short input-output pair.

• If it can, start with SFT.
• Mid-training earns its keep if the task depends on the model taking in how long documents are organized, moving through multi-stage states, or handling a context mix unlike what it saw before.
• If the missing information changes regularly, neither fits, and retrieval or tools are the answer.

For INV-22041, valid SQL JSON is an SFT target. Curated long trajectories may help the agent recover across a 30-step investigation with several failed hypotheses. Today's sales figure remains runtime data.

Parameter-efficient tuning: LoRA, QLoRA, DoRA and prefix tuning

Parameter-efficient fine-tuning ought to be the enterprise default, since it keeps one shared base model apart from small, versioned behavior deltas.

LoRA

LoRA freezes a weight matrix W and learns a low-rank update. The adapted matrix equals W plus the product of two small matrices B and A, scaled by alpha divided by the rank r.

Suppose W maps 4096 inputs to 4096 outputs and the rank is 16. The dense matrix then holds roughly 16.8 million parameters, while the two LoRA matrices have just 131,072 trainable parameters for that projection (16 × 4096 plus 4096 × 16). That's about 0.78% of the dense matrix.

Three settings define what the adapter can do: the rank sets its capacity, alpha divided by rank scales the update, and the target modules decide where the model may adapt. All three are part of the model's identity and must be registered.

Why a low rank can be enough

A dense update lets every input-output direction change on its own. LoRA bets that the task-specific update lives in a far smaller intrinsic subspace. Rather than learn d_out × d_in values, it learns only r(dᵢ + dₒ), with dᵢ the width going in and dₒ the width coming out.

For a square 4096 by 4096 projection at rank 16, the numbers are:

• a dense update would need 16,777,216 parameters;
• the LoRA update needs 131,072;
• the trainable fraction is 0.0078125, or 0.78125%.

Those figures describe one projection. A real adapter targets many projections across many layers, so count the trainable parameters from the instantiated model and don't extrapolate from a single matrix.

The scaling term isn't cosmetic

With alpha at 32 and rank 16, the LoRA multiplier is 2. Raise the rank to 64 while alpha stays at 32 and the multiplier falls to 0.5. The learned matrices and the optimization dynamics change even though the dataset and base model haven't.

Some implementations use rank-stabilized scaling, proportional to alpha over the square root of the rank, in place of alpha over rank. That alters comparisons between ranks. Record the scaling convention, the library version, the initialization and the dropout.

Where to attach adapters

The attention projections each play a different part:

• q_proj shapes the question the current token puts to the context;
• k_proj governs how earlier tokens signal what they contain;
• v_proj sets the content actually pulled from them;
• o_proj decides how attention's output is written back into the residual stream.

MLP projections shape feature transformation and can matter for domain behavior. Tuning only query and value projections is a common starting point, not a rule. An ablation matrix should vary target modules and rank, and track trainable count, task gain, forgetting, latency and artifact size.

Composing adapters isn't automatically additive

Two adapters that work well alone can still clash once both are active. Their deltas may push the same layers in conflicting directions, shift activation statistics, or rely on different system packs. Treat a composition as a fresh recipe: the composed weights are the base plus each adapter's delta times its scale sᵢ. That equation looks simple, but updated projections pass through attention, normalization and MLP layers nonlinearly, so test the composition exactly as it's served.

QLoRA

QLoRA stores the frozen base in 4-bit form while gradients flow only into LoRA adapters. It leans on NF4, double quantization of the quantization constants, and paged optimizers that soak up memory spikes.

Two distinctions head off production mistakes:

• QLoRA is a training-memory technique. It doesn't define the serving quantization on its own.
• An adapter alone doesn't show that the quantized deployment behaves equivalently. Repeat the quality, latency, leakage and tool-use evaluations on the build you actually serve.

Why NF4 and double quantization matter

Ordinary 4-bit integers split a range into evenly spaced values. Pretrained weights are often close to normally distributed, so even spacing uses its representational levels inefficiently. NF4 places the values to fit that distribution better.

Blockwise quantization gives every block its own scale, a constant that has to be stored. Double quantization squeezes those constants as well, which trims metadata overhead across a big model.

As a rough memory estimate for frozen weights, multiply parameter count by bits per weight, divide by 8, and add metadata. Stored ideally at 4 bits, a 7-billion-parameter base needs about 3.5 GB of raw weights (7B × 4 / 8). Scales, buffers, temporary dequantization, adapters, gradients, optimizer state, activations and framework overhead come on top, so a claim that a 7B model fits in 3.5 GB promises nothing about training memory.

QLoRA saves memory on the frozen base weights and on adapter optimization. Activation memory still scales with batch size, sequence length, layer width and checkpointing policy, which means long-context training can remain activation-bound even with 4-bit weights.

Paged optimizers

Activation checkpointing and long sequences can cause temporary memory spikes. Paged optimizers use managed memory to shift optimizer pages between CPU and GPU as needed. That can prevent an out-of-memory failure, though paging may cut throughput. Measure wall-clock time and host-memory pressure, and don't treat a successful allocation as a successful optimization.

DoRA

DoRA splits a weight into a magnitude and a direction and applies the low-rank update mainly to the direction. The adapted weight is a learned magnitude m times the updated matrix W + ΔW, normalized column by column. Here ΔW is the LoRA-style update. The intuition is that magnitude and direction play different roles, and separating them can recover quality at modest ranks for some added complexity.

The normalization denominator fixes each updated direction, and m then controls its magnitude. That brings DoRA closer to the degrees of freedom available in full fine-tuning, at the price of extra parameters and operations. The enterprise question is empirical: does the quality gain survive the added training, serving and registry complexity?

Prefix and prompt tuning

Prefix tuning learns continuous virtual tokens that steer attention and leaves the base weights alone. That makes it compact and good for narrow control, though it expresses less than LoRA when the target is tool grammar or multi-step behavior.

Sidecar or merge?

Sidecar adapters suit tenant or task isolation, fast rollback, A/B routing and auditability. Merging is appropriate only for a release meant to stay immutable, after the exact merged checksum has been evaluated and registered.

Sidecars also allow routing by declared purpose, so a Reporting adapter can't quietly activate during Data Sufficiency. The router's decision belongs in the trace, along with the adapter ID, scale, base checksum and reason.

Merging cuts runtime adapter overhead and may simplify deployment, though it blurs the clean line between base and behavior delta. Preserve the unmerged lineage, checksum the merged model, and never merge into an unpinned "latest" base. A compact training configuration looks like this:

from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
lora = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM",
)
trainer = SFTTrainer(
    model=base_model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    peft_config=lora,
    args=SFTConfig(
        output_dir="runs/inv22041-sft-v3",
        assistant_only_loss=True,
        packing=True,
        max_length=4096,
    ),
)
trainer.train()

The notable lines include more than r=16 and lora_alpha=32. assistant_only_loss=True guards the loss boundary, the pinned target modules define the adapter's scope, and packing has to be compatible with the chat template's masks.

Offline preference optimization

SFT tells the model to imitate. Preference optimization tells it to move toward one option among plausible alternatives. Take two reports on the same investigation, both grammatical and statistically sound. The chosen one cites the governed evidence, names its assumptions and stops there. The rejected one tacks on an unsupported causal story. That pair carries a stronger signal than yet another gold answer would.

The Bradley-Terry view

Pairwise preference models often start from one statement: the probability that response yʷ beats response yˡ for input x is the sigmoid of the reward gap, σ[r(x, yʷ) − r(x, yˡ)]. Here yʷ is the chosen response, yˡ the rejected one, and r an implicit or explicit reward. Only the difference between the two rewards matters.

DPO

DPO optimizes a policy directly on preference pairs while tying it to a reference policy. The loss is the negative log-sigmoid of beta times a margin, and the margin is how much the policy's log-probability for the chosen response rose over the reference's, minus the same quantity for the rejected response. In the notation, πθ is the policy under training, π₀ is the frozen reference model, and β sets how strong the reference-relative preference margin is.

The reference checkpoint and beta count as release metadata. Changing either one changes the objective even when the dataset is identical.

A worked DPO calculation

Assume the current policy and the reference give these sequence log-probabilities:

• current policy, chosen report: −8.0;
• reference, chosen report: −9.0;
• current policy, rejected report: −7.5;
• reference, rejected report: −7.0.

Relative to the reference, the policy improved on the chosen answer by −8.0 − (−9.0) = 1.0. On the rejected answer the shift is −7.5 − (−7.0) = −0.5. The DPO margin is 1.0 − (−0.5) = 1.5. With beta = 0.2, beta times the margin comes to 0.3, and the loss is −log σ(0.3), roughly 0.554.

The policy earns credit because it raised the chosen response relative to the reference and lowered the rejected one. A negative margin would give a larger loss, and the gradients would reverse that ordering.

The calculation also shows why the reference matters. DPO doesn't just ask which response has the higher raw probability. In this example the rejected response still has the higher current log-probability (−7.5 against −8.0), but the policy moved in the right direction relative to the reference.

Length is part of the signal

A sequence log-probability is a sum over tokens, so longer sequences tend to pile up more negative log-probability. Dataset construction and objective variants deal with this in different ways.

If chosen reports are consistently shorter than rejected ones, the model may learn brevity in place of evidence discipline. If chosen reports run longer, it may learn verbosity. Stratify preference pairs by length difference, and check whether win-rate gains hold up under length-controlled evaluation.

SimPO uses the average log-probability, which normalizes for sequence length, and adds a target reward margin. That lowers dependence on a separate reference model but brings its own margin and scaling choices. Reference-free doesn't mean hyperparameter-free.

Preference strength and disagreement

A binary pair says only that y_w won. It doesn't say whether the win was obvious or narrow. Where possible, preserve rater votes, confidence, rubric dimensions and disagreement.

A 3-2 rater split needn't carry the same training weight as a unanimous 5-0 verdict. Down-weighting ambiguous pairs has a cost too, because valid minority preferences can disappear. Check by user cohort, language, geography and task class before picking a weighting rule.

Reference drift

Say preference training is meant to start from sft-v3 but the DPO runner quietly loads sft-latest. The mathematical reference has changed. Store both the initialization policy and the frozen reference policy used in the loss. They're usually the same, though they needn't be, and reproducibility demands both IDs and checksums.

The neighboring methods

• IPO swaps the unbounded push of the logistic preference loss for a finite, margin-style target, which helps when deterministic labels would otherwise let DPO overfit.
• KTO learns from examples labeled desirable or undesirable independently, so paired outputs aren't required.
• ORPO folds instruction learning and an odds-ratio preference penalty into one objective, with no separate reference model.
• SimPO is reference-free, using length-normalized sequence log-probability plus a target margin.

No method makes leakage tests or an SFT baseline unnecessary. Preference training can raise win rate while degrading calibration, response length, refusal boundaries or minority slices.

Choosing within the DPO family

Make the choice with a decision record and not by trend:

• pick DPO for well-formed pairs and a stable reference model that's affordable to run;
• try IPO when deterministic or low-noise preferences are driving DPO toward excessive margins;
• pick KTO when feedback arrives as independent good-or-bad outcomes and pairing would be artificial;
• try ORPO if a single phase combining SFT and preference learning simplifies operations;
• try SimPO if memory for a reference model is the tightest limit and rewards normalized by sequence length fit the task.

Start every candidate from one SFT baseline and give it the same data split, evaluation suite and compute budget. If the setups differ, you're comparing pipelines and not objectives. A minimal DPO setup looks like this:

from trl import DPOConfig, DPOTrainer
trainer = DPOTrainer(
    model=sft_model,
    ref_model=reference_sft_model,
    train_dataset=preference_dataset,
    args=DPOConfig(
        output_dir="runs/inv22041-dpo-v1",
        beta=0.2,
        max_length=4096,
        max_prompt_length=2048,
    ),
)
trainer.train()

The reference has to be the checkpoint written in the run manifest, and never whatever the model registry currently calls "latest."

Classic RLHF: reward model, PPO and RLAIF

Classic RLHF runs three models together. The policy writes candidate answers, a reward model grades them, and PPO adjusts the policy, holding back any movement big enough to destabilize it.

PPO relies on a clipped surrogate objective. Compactly, it takes the expected minimum of the probability ratio times the advantage and the clipped ratio times the advantage, then subtracts a KL penalty that measures divergence from a reference. The symbols mean the following:

• ρₜ is the probability ratio between the new and old policies;
• Aₜ estimates whether the sampled action beat expectations;
• ε caps how large a policy update may be;
• the KL penalty holds the policy near a reference.

Pick this stack when quality is truly subjective, there are enough comparisons to fit a reward model, and online policy optimization justifies the added operational complexity.

Training the reward model

A reward model usually applies the same pairwise Bradley-Terry loss: the negative log-sigmoid of the learned score for the chosen response minus the score for the rejected one. The scorer rφ is a learned scalar function with parameters φ, trained to rank chosen responses above rejected ones.

The absolute value of a reward has no meaning of its own. A score of 7 isn't "seven units good," and only comparisons inside the calibrated operating distribution are informative. The scale of rewards can wander from one checkpoint, dataset or prompt class to the next.

Treat a reward model as a ranking system and measure:

• accuracy on held-out pairs;
• calibration by score difference;
• agreement with human labels by slice;
• resistance to adversarial inputs;
• sensitivity to length, formatting, citation count and identity terms;
• behavior out of distribution.

Reserve a gold set that never trains the reward model and never guides the policy's optimization. Otherwise the policy can overfit its evaluator while the dashboard shows progress.

PPO clipping with real numbers

Say the old policy gives an action probability 0.20 and the updated policy gives it 0.25. The ratio is 0.25 / 0.20 = 1.25. With ε = 0.2, clipping caps the ratio at 1.20. For a positive advantage of 2:

• the unclipped term is 1.25 × 2 = 2.50;
• the clipped term is 1.20 × 2 = 2.40;
• PPO takes the smaller of the two, 2.40.

The update still reinforces the action, but it can't claim the full objective gain from a big jump in probability.

When the advantage is negative, taking the minimum keeps the optimizer from gaming the clipping in the opposite direction. That cautious step is the reason the old-policy snapshot and rollout version stay frozen for the entire batch.

The KL controller

The KL term discourages the policy from drifting too far from a reference. The adjusted reward is the learned reward minus a coefficient βᴋʟ times the KL divergence between the policy and the reference. If βᴋʟ is too small, the policy may exploit the reward model, turn repetitive or lose language quality. If it's too large, learning stalls. Some systems adapt beta to hold KL within a target range.

Track KL by token position and task slice and not only as a global average. A modest average can conceal a big shift on refusals, tool calls or minority languages.

The actor-critic bill

Classic PPO typically calls for six pieces. There's the actor that's being trained, plus a frozen snapshot of its previous version, a reference policy that anchors the KL term, and a value model acting as critic. A reward model and a rollout engine round out the list. They may share weights or infrastructure while keeping separate identities. Offline preference methods and critic-free group methods appeal because they delete parts of a costly, failure-prone distributed system.

Don't pick PPO simply because it's the canonical RLHF method. It earns its place when optimizing against a learned, subjective reward is worth the extra machinery to run and maintain.

RLAIF swaps in AI judgments, guided by a written constitution, for some or all human preference labels. What it adds is scale, never ground truth. The governed data contract has to cover the judge's prompt and constitution, which judge version ran, the slices where judges disagree, and when a human steps in.

Reward hacking is the most dangerous failure: the policy finds a surface feature that scores well without serving the real goal. An enterprise reward model may pick up that longer reports, extra citations or frequent hedging phrases mean "better." Adversarial probes and direct task metrics belong outside the learned reward.

RLAIF and constitutional judgments

A constitution turns principles into a judging protocol. For INV-22041 it could instruct the judge to rank authorization first, then citation support, statistical validity, calibrated uncertainty and concision.

Judge-based labels add three further artifacts to track:

• the constitution and the judge prompt;
• the judge model and its decoding configuration;
• the adjudication policy for disagreement or uncertainty.

Position bias shows up when you vary ordering. Swap chosen and rejected, randomize neutral identifiers, and add human-reviewed sentinel pairs. A judge that changes its verdict when the order flips can't be trusted as a source of preference truth.

Reinforcement learning with verifiable rewards

If success can be checked mechanically, a learned reward model may not be needed at all. SQL can be parsed and run in a cage. Code can be executed against tests. A statistical conclusion can be compared with expected results and assumptions.

GRPO

GRPO samples a group of candidate responses to one prompt and normalizes their rewards. A response's advantage is its reward minus the group mean, divided by the group's standard deviation plus a small stabilizer ε.

Imagine eight candidate SQL plans with rewards that, sorted from worst to best, read −0.3, −0.2, 0, 0, 0.5, 0.8, 0.8 and 1.0. Their mean is 0.325 and the population standard deviation is roughly 0.47. Each candidate is pushed up or down relative to its sampled peers, which removes the need for a separately trained critic, though the method is only as sound as its reward.

For the best trajectory the normalized advantage is roughly (1.0 − 0.325) / 0.47, or about 1.44. For the worst it's (−0.3 − 0.325) / 0.47, about −1.33.

The exact numbers depend on whether an implementation uses the population or the sample variance, and on how rewards are preprocessed. Pin these details, because they change the gradient scale.

If every sample gets a reward of 1, the standard deviation is zero and no relative signal exists. The epsilon prevents a division by zero, but it can't manufacture information to learn from. Dynamic sampling or a harder curriculum is the remedy.

For INV-22041, a composite reward might demand that:

• the JSON is valid against the schema;
• the tenant predicate is present;
• the SQL is read-only;
• the query runs within a cost limit;
• the returned data suffices for the declared test;
• the report cites only evidence released to the agent.

A naive weighted sum lets excellence on one dimension compensate for a critical violation. Take a reward made of 0.3 times correctness, 0.2 times evidence quality, 0.2 times SQL validity and 0.3 times style. A cross-tenant query could still score positively if the answer and the style were good. Safety and authorization therefore need hard gates. In the gated form the reward is zero whenever authorization failed or a protected-data leak occurred. Otherwise it's a weighted sum of correctness, tool and evidence rewards minus a weighted cost term.

In riskier settings a violation can earn a negative terminal reward in place of zero. What matters is non-compensation, since fluency can never pay for unauthorized access.

Keep reward components separately observable. The optimizer needs one scalar, but a dashboard that plots correctness, authorization, tool validity, evidence quality, cost and length on their own makes reward hacking diagnosable.

Reward normalization can hide business magnitude

Group normalization cares only about the relative order within one prompt. If every candidate is terrible and one is slightly less terrible, that one can receive a positive advantage. Absolute release floors are still needed.

The opposite case is a group where every candidate is correct. It teaches little even though it demonstrates healthy production behavior. Keep those prompts in evaluation, and remove or down-weight them for relative-policy optimization only if the sampling policy is stated explicitly.

Policy ratios and clipping

GRPO-style objectives usually retain PPO's importance ratios and clipping. Per token, the ratio divides the new policy's probability by the old policy's, and the objective is the expected minimum of ratio times advantage and clipped ratio times advantage, minus a KL term. The advantage can be defined per sequence while ratios are applied per token, a mismatch that partly motivated sequence-level successors such as GSPO.

DAPO: steadier training on long reasoning

DAPO bolts on four practical mechanisms. Clipping bounds are decoupled, groups get sampled dynamically, the policy gradient is handled per token, and samples that run too long are dealt with explicitly. Dynamic sampling discards groups whose rewards are all identical, since those carry no relative signal.

Vanilla GRPO may suffer entropy collapse, long-chain instability, or batches full of all-correct or all-wrong groups, and that's when DAPO helps. Its recipe reads best as four engineering responses to observed training failures:

1. Clip-Higher. The upper and lower clipping bounds are decoupled, giving useful low-probability tokens more room to grow without inviting symmetric instability everywhere.
2. Dynamic sampling. Prompts whose sampled group is entirely correct or entirely wrong get filtered out, since their normalized advantages offer little contrast.
3. Token-level policy-gradient loss. Long responses count at token granularity and don't get diluted by sample-level averaging choices.
4. Overlong reward shaping. Length penalties phase in gradually, avoiding a hard boundary that would create unstable gradients near the maximum length.

Each mechanism answers a different symptom. Don't adopt the acronym as one indivisible package without measuring which failure your workload actually has.

In an enterprise SQL agent, dynamic sampling can concentrate training on queries where only some plans bind the right tenant. Overlong shaping stops the model spending thousands of reasoning tokens ahead of a simple query, and Clip-Higher helps when useful recovery actions begin with very low probability.

GSPO: the sequence as the unit

GSPO computes importance ratios and clipping across the whole sequence and not independently per token. That fits sequence-level rewards better, and reports credit it with steadying RL on very large Mixture-of-Experts models.

The lesson isn't that GSPO replaces GRPO. It's that the unit of optimization should match both the reward and the architecture. When the business outcome is scored per sequence and the policy is a Mixture-of-Experts model, GSPO deserves a trial.

Roughly speaking, GSPO builds a sequence likelihood ratio by averaging the per-token log-probability differences over the response length and exponentiating the result, which returns the value to ratio space. The objective then clips that sequence-level ratio and applies the group-derived advantage at the same level. Since the reward judges the whole trajectory, the optimization unit now matches the judgment unit.

Small routing changes between tokens in an MoE model can make token-wise ratios noisy, so aggregating across the sequence may be steadier, at the price of less localized credit. Judge convergence, entropy, router stability, reward and downstream quality together and don't rely on reward alone.

RLOO

REINFORCE Leave-One-Out takes the other samples in a prompt group as each sample's baseline. It's a critic-free policy-gradient estimator and a handy yardstick when comparing group-based methods.

With G samples, sample i gets a baseline equal to the mean reward of the other G − 1, and its advantage is its reward minus that baseline. There's no division by the group's standard deviation as in z-score normalization. The estimator is simple and unbiased under the REINFORCE construction, though reward scale carries straight into gradient scale.

Take rewards of 1.0, 0.8 and 0.2. For the first sample the leave-one-out baseline is (0.8 + 0.2) / 2 = 0.5, leaving an advantage of 0.5. The third sample's baseline is (1.0 + 0.8) / 2 = 0.9, which yields −0.7.

RLVR and reinforcement fine-tuning

The wider pattern is reinforcement learning with verifiable rewards. Sample trajectories, run deterministic or tightly specified graders, and reinforce whatever passes.

Outcome rewards check whether the final answer is right, while process rewards score intermediate steps. Process supervision can localize a failure, but step graders cost a lot and may over-constrain valid reasoning. In enterprise agents, process rewards earn their keep at observable interfaces (tool selection, authorization, argument construction, evidence release and hand-off) and never on hidden chain-of-thought.

Outcome reward against process reward

An outcome-only SQL reward could be a simple indicator that equals 1 when the result matches the expected one and 0 otherwise. That's objective and inexpensive to compute, yet it lets a model arrive at the answer by way of an unsafe or expensive query.

Process-aware rewards add observable invariants, summing six terms. They cover whether the schema is valid, whether the tenant is bound, whether the query stays read-only, what it costs, whether the outcome is right and whether evidence was properly released. Each term should be a tested function of visible artifacts. Scoring should cover the SQL syntax tree, the authorization decision, the execution plan, the returned dataset, the statistical test record and the citations, and never private chain-of-thought.

Grader tests are model tests

Build a grader test matrix before any RL run:

• a known-good trajectory passes;
• the wrong tenant fails;
• a tenant predicate hidden in a subquery is read correctly;
• a SQL comment can't spoof a required predicate;
• an empty result is told apart from an execution error;
• a high-cost Cartesian join fails;
• a fabricated citation fails even when the final number is right;
• a grader timeout fails closed;
• malformed grader output can't become a positive reward.

After each grader change, replay historical trajectories. A change in reward version starts a new experiment lineage even if the policy code is unchanged.

Thinking has a price

Correctness alone can train wasteful reasoning, so add a bounded cost term: the reward is correctness minus lambda times the thinking cost Tₕ. That cost can be reasoning tokens, milliseconds or normalized compute, and λ expresses what extra deliberation costs. Apply the penalty only after the correctness and safety floors, or the model learns to answer quickly and wrongly.

Say two correct trajectories both earn a base reward of 1.0, one using 400 reasoning tokens and the other 1,000. With token cost normalized by 1,000 and lambda = 0.1, the short one nets 0.96 (a 0.04 deduction) and the long one 0.90 (a 0.10 deduction).

So the shorter correct trajectory comes out ahead. Still, an incorrect 100-token answer should fail the correctness gate and shouldn't collect −0.01 and compete with correct answers.

Latency and token count correlate without being identical. Parallel tool calls, cache hits, batch scheduling and the model architecture can move wall-clock time while reasoning tokens stay put. Choose whichever cost measure mirrors the production constraint.

Multi-turn agent RL

Model decisions, tool results, memory reads and agent hand-offs all sit inside an enterprise trajectory. Store its state transitions, and score the final business contract in addition to the local invariants.

For instance, Enrichment may earn no direct reward for asking a clarifying question. Its value shows up when Data Sufficiency writes the right query and Reporting avoids a false causal claim. Credit assignment mustn't erase that dependency.

The trajectory is a typed event log

Represent the rollout as more than a chat transcript. Each transition should carry:

• state and observation identifiers;
• the acting agent and its policy version;
• the available tool set and authorization scope;
• the selected action and its raw arguments;
• the policy decision and the tool result;
• memory reads and writes;
• the hand-off payload;
• timestamps, cost and the terminal outcome.

This form supports replay, off-policy analysis and precise reward assignment. It also stops the training pipeline from treating tool observations as if the assistant had generated them.

Hierarchical rewards

For a four-stage investigation, define local contract rewards plus a terminal business reward. The total reward is the terminal reward R★ plus a weighted sum of four stage rewards R₁ to R₄, one each for Enrichment, Data Sufficiency, Hypothesis Testing and Reporting, with weights γ₁ through γ₄.

Local rewards speed up learning, and they can also conflict. Enrichment may maximize clarification quality by asking far too many questions. Reporting may optimize brevity by leaving out a limitation. The terminal reward and the hard invariants must outweigh local convenience.

Credit should follow causal responsibility. If Enrichment hands Reporting a wrong entity mapping and Reporting faithfully reports the result it received, penalizing Reporting alone teaches the wrong thing. Keep each stage's output and judge the earliest contract violation.

Good practice: train in the smallest environment that still preserves the real contract, with sandboxed tools, deterministic fixtures, versioned graders, strict timeouts and no production credentials.

Distillation: transferring capability and not copying a model

Distillation turns an expensive teacher's behavior into a smaller or easier-to-deploy student.

Logit distillation

The student is trained to match the teacher's softened token distribution. The loss is the squared temperature times the KL divergence between the teacher's softmax and the student's softmax, both computed on logits divided by the temperature T. The squared factor keeps the gradient scale intact. A higher temperature exposes the relative probabilities beyond the top token, the "dark knowledge" that hard labels leave out.

What temperature changes

Imagine a teacher with logits 8, 4 and 2 for three candidate tokens. Under T = 1 the softmax puts nearly all the mass on token one. With T = 2 the effective logits drop to 4, 2 and 1 and the distribution softens, so the student learns that token one wins and also that token two is likelier than token three.

As temperature rises, gradients from the secondary alternatives become more informative. Push it too high and the distribution flattens toward uniform, losing useful structure. Tune it on held-out transfer quality.

The squared factor compensates for gradients shrinking when logits are divided by the temperature. Drop it and the balance between the distillation loss and the hard-label loss shifts.

Combining hard and soft targets

A common objective mixes two terms: a weight on the SFT loss plus a weight on the distillation loss. Hard labels keep the known correct output, and soft teacher probabilities carry over relative preferences. The two weights set the balance.

For tool calls, hard schema correctness may deserve more weight than the teacher's uncertainty. For natural-language explanation, the teacher distribution may carry richer behavior. Weights can differ by token class, but that added complexity has to stay testable and documented.

Sequence and on-policy distillation

Sequence distillation trains on teacher-generated outputs. It's simple, but the student only sees the teacher's state distribution.

On-policy distillation lets the student generate, then has the teacher supervise the states the student actually reaches. That helps especially with agents, where small early deviations produce new downstream tool states.

When the goal is to distill reasoning, carry over what can be seen: how problems get solved, the tool traces, and outcomes that can be verified. Hidden reasoning text shouldn't become a production dependency.

Filters for sequence distillation

Teacher-generated output is synthetic training data, and it deserves the same controls as human-written data:

• review for authorization and licensing;
• scan for PII and secrets;
• run tools in a sandbox;
• verify deterministically wherever possible;
• deduplicate against evaluation;
• track provenance back to the teacher, the prompt, the decoding settings and the source context.

Best-of-N selection can raise synthetic quality, but it narrows diversity and can amplify the selector's biases. Keep generation and selection statistics so the student's data distribution can be reconstructed.

The on-policy distillation loop

On-policy distillation cycles through five steps:

1. The student generates from its current policy.
2. The teacher scores those states or supplies token distributions for them.
3. Unsafe or invalid trajectories are filtered out.
4. The student updates on the remaining states.
5. Evaluation checks capability and drift.

The loop aims at the student's real errors. It costs more than static teacher data and can inherit instability from a student distribution that keeps moving. Pin the teacher's identity and snapshot the student's rollouts at each iteration.

Distillation isn't speculative decoding

Distillation trains a student to approximate a teacher. Speculative decoding runs a smaller drafter at inference time and has the target model verify the proposed tokens. A distilled student can serve as the drafter, but the two techniques chase different goals.

Distillation makes a model cheaper to run on its own. Speculative decoding only speeds up a target model when many draft tokens get accepted and verification is efficient. Judge them separately.

The enterprise distillation contract

Pin down each of these:

• the teacher model and API version;
• generation parameters and the tool environment;
• the prompt or system-pack version;
• filtering and adjudication rules;
• the student base, tokenizer and adapter lineage;
• safety evaluations for both teacher and student;
• data-use rights covering teacher outputs.

A smaller model inherits failure modes too. Re-run the privacy, citation, authorization, tool-use and adversarial suites on the student, and don't assume the teacher's compliance carries over.

What frontier models changed in training and fine-tuning

It isn't parameter count that explains recent frontier models. Their pipelines grew past one supervised fine-tuning stage. The strongest ones blend several ingredients: data fed in a deliberate order, reinforcement learning, rewards that can be verified, synthetic trajectories, and transfer from teacher to student. Each plugs a different gap that imitation alone leaves open.

Fine-tuning is no longer one operation. It's a curriculum that carries the model from imitation to exploration, then verification, then alignment, and finally deployment.

DeepSeek-R1

DeepSeek-R1 put reinforcement learning at the center of reasoning adaptation. R1-Zero started from a pretrained base model and applied RL without first teaching it any human-written chains of thought. The experiment tested whether reasoning could emerge from outcome optimization and not from copying curated reasoning traces.

The central method was Group Relative Policy Optimization (GRPO). A policy generates a group of candidates for one prompt and each one gets a reward. Advantage comes from comparing against the group, meaning reward minus group mean over group standard deviation plus ε, with no separately trained value model involved. Responses that beat their peers get a positive advantage and weaker ones a negative advantage. Dropping the critic trims memory and compute while keeping a useful learning signal.

GRPO worked especially well where correctness could be checked automatically: answer verification for mathematics, compilation and tests for code, and format validation for structured responses. The model learned from objective outcomes, with no human preference label needed for every reasoning step.

R1-Zero showed that behaviors such as verification, backtracking and spending more computation on hard problems could emerge through RL. It also exposed the limits of pure outcome training. Reasoning could become hard to read, inconsistent in language, or tuned toward the reward without producing a polished response.

The complete DeepSeek-R1 pipeline answered those limits with a staged curriculum. It starts with readable cold-start examples, moves to reasoning RL, then rejection sampling, then mixed reasoning and general SFT, and ends with alignment RL. Cold-start data fixed a readable reasoning style, reasoning RL extended capability past imitation, rejection sampling retained the better self-generated trajectories, SFT on reasoning and general data consolidated the behavior while keeping broad usefulness, and the closing RL stage improved helpfulness and safety.

GRPO alone wasn't the novelty. What counted was how supervised learning, verifiable RL, self-generated data and final alignment were sequenced.

DeepSeek next fine-tuned smaller dense models on successful reasoning trajectories. Part of an expensive reasoning ability thus moved into models that are easier to deploy, and the students learned from demonstrations holding richer problem-solving structure and not just final answers.

Qwen3 and mode fusion

Qwen3 handled reasoning and direct response as behaviors to be learned together. Post-training began with a long-chain-of-thought cold start, went on to reasoning-focused RL, fused reasoning and non-reasoning behaviors, and then ran broader RL for general alignment.

Mode fusion is the interesting training idea. One policy learns both behaviors, where the alternative is a long-reasoning checkpoint plus a separate brief-answer checkpoint. It works out when an extended reasoning trajectory is warranted and when a direct reply will do, so reasoning ability survives without pushing every task into the same verbose style.

GSPO and the granularity of the update

GSPO (Group Sequence Policy Optimization) changed how fine-grained the policy update is. Token-level methods typically compute and clip importance ratios separately per token even when the reward covers the entire response, and that can make updates inconsistent inside one generated sequence.

GSPO forms the importance ratio over the sequence and clips the sequence as one optimization unit. In simplified terms, it averages the log probability ratio between the updated policy and the policy that generated the trajectory across the response length, then exponentiates. Averaging across the sequence gives one response-level signal, which matches the unit that received the reward.

It pays off most when reinforcement learning turns unstable because token-level probability shifts are noisy. The novelty is lining up three things that often don't match: the generated object, the reward unit and the optimization unit.

Kimi K2: stability and agentic trajectories

Kimi K2 brought in training-stability methods that made very large optimization runs more dependable. MuonClip pairs Muon-style matrix optimization with controls that stop weights from growing unstably. QK-Clip watches attention behavior and selectively constrains troublesome query and key projections when their logits get extreme.

These are pretraining innovations and no downstream fine-tuning techniques, yet they matter here because post-training quality rests on the checkpoint it inherits. A small adaptation dataset can't efficiently repair a base model damaged by optimizer instability or attention-logit explosion.

Kimi K2's post-training leaned on agentic trajectories too: coding, tool invocation, observation handling and multi-step task completion. Training uses sequences of actions with environmental feedback instead of only prompt-answer pairs. A typical path runs from task to tool selection, argument generation, execution feedback, error recovery and a verified outcome.

Parts of such trajectories can be checked mechanically, thanks to code that runs, tool responses and signals that a task finished. The loop can then reward more than a convincing final answer. It can check whether the model picked the right capability, produced valid arguments, reacted properly to observations and finished the task.

Teacher-student pipelines at Meta

Meta's Muse Spark and Muse Glimmer illustrate a fuller teacher-student pipeline. The teacher may produce final answers, intermediate reasoning, grounded multimodal readings, tool decisions and recovery trajectories, but none of that becomes training material until it's been filtered for correctness, safety, provenance, diversity and relevance.

Muse Glimmer's transfer chains seven adaptation steps. A teacher first generates traces, which pass through governed filtering. The student then goes through logit distillation, mid-training and supervised fine-tuning, followed by on-policy knowledge distillation, with reinforcement learning at the end.

• Logit distillation transfers the teacher's probability distribution and not just its chosen token. That carries information about plausible alternatives and relative confidence that hard-label training discards.
• Mid-training shows the student long-context and agentic data ahead of narrow instruction tuning, so those skills end up in its broader representation and don't sit as a thin layer added last.
• On-policy knowledge distillation repairs a core weakness of plain distillation. Trained on teacher trajectories alone, a student learns the teacher's states, yet at inference it encounters states created by its own flawed decisions. If the student produces the trajectory and the teacher advises on those states, the mismatch shrinks.
• Reinforcement learning then strengthens behaviors that have measurable outcomes.

Taken together, the steps make distillation a multi-stage capability-transfer program instead of one compression pass.

The frontier recipe is now visible: supervised data to set behavior, verifiable RL to pass imitation, synthetic trajectories for coverage, an optimization unit that matches the reward's granularity (the whole sequence), and distillation on the student's own states to hand capability over to models that can be deployed.

No single model name is the lasting innovation. The training system is: the order of the data, the verification of outcomes, the way the policy explores, the control of unstable updates, and a final transfer that doesn't assume a smaller student can reproduce its teacher perfectly.

Training is a release

A run that succeeds still isn't a deployable artifact. What ships is a full recipe. It spans the base model and its tokenizer, either the adapter or merged weights, the system pack and tool schemas, the memory contract and retrieval policy, and finally quantization, runtime and safety policy.

Registry identity

At a minimum, register:

• checksums for the base and reference models;
• digests for the dataset and its transforms;
• the split manifest and the contamination report;
• the objective, optimizer, seed, hyperparameters and target modules;
• the code and container digests;
• hardware and numerical precision;
• versions of the grader, reward model, judge and policy;
• the evaluation report, approvals, owner, expiry date and rollback target.

Evaluation needs floors and comparisons

Apply two gates:

1. Absolute floors. Leak rate is zero on protected suites, tool schema validity clears its threshold, tenant binding is 100%, and the refusal and safety floors pass.
2. Relative improvement. A candidate has to outperform the current production recipe on the target defect while all floors still hold.

Evaluate the whole runtime and not only the bare model. A stronger adapter can regress once a new quantization, chat template, retrieval policy or tool schema is applied.

Don't collapse enterprise quality into one average

Say Mₖ is the metric for evaluation slice k. A weighted score S sums each slice's weight times its metric, with the weights totaling 1, which condenses results nicely. Yet a weighted average permits compensation, so a big gain on common English reporting prompts can conceal a serious step backward in tenant isolation or in languages with little data.

Gate lexicographically in this order:

1. hard safety, privacy and authorization floors;
2. capability-preservation floors;
3. improvement on the target task;
4. cost and latency constraints;
5. an aggregate ranking among the candidates that cleared every earlier gate.

This mirrors the reward-design principle: non-negotiable contracts can't be bought back with style or throughput.

Confidence intervals and small gains

Sampling noise can explain an observed accuracy difference. For a binary metric with proportion p̂ over n independent items, the standard error is roughly the square root of p̂(1 − p̂) / n. With p̂ = 0.90 and n = 100 it's about 0.03, which makes a one-point gain unconvincing.

Candidates answer the same prompts, which makes paired tests usually stronger than treating the results as independent. Bootstrap the per-item differences or use a suitable paired test, and report confidence intervals along with point estimates.

LLM-as-judge evaluations need extra controls: randomized order, pinned judge versions, human calibration and tracking of judge disagreement. A narrow lead from one judge prompt doesn't make a release claim.

The forgetting suite

The target gain mustn't erase general capabilities. Keep a fixed suite covering:

• instruction following;
• where safety and refusals draw the line;
• behavior across languages;
• picking the right tool and formatting its call;
• finding facts in long contexts;
• calibrated confidence and knowing when to abstain;
• neighboring domain tasks that weren't the target;
• adversarial and privacy probing.

Report deltas against the exact base or production recipe. When a capability floor fails, inspect the training data, rank, learning rate, epochs, replay mixture and objective before reaching for more target examples.

Two quantization gates

When QLoRA or later serving quantization is involved, evaluate twice:

1. The training-artifact gate uses the base plus adapter in the evaluation precision from development.
2. The serving-artifact gate uses what production runs: the exact merged or sidecar build, runtime kernels, quantization, tokenizer and chat template.

The first isolates whether training taught the behavior. The second shows whether deployment kept it.

Canary and rollback

Canaries can be sliced by bounded traffic, tenant class, request type or shadow mode. Watch how often tasks succeed and humans step in to correct, how often policy refuses, tool errors, spend in tokens and latency, plus any drift in rewards or in the input distribution.

Rollback has to restore the prior recipe, which means more than the prior adapter. If the new system pack, tokenizer or tool schema is still active, the system hasn't been rolled back.

The arithmetic of canaries

Say a critical failure has historically occurred at rate p. Over n independent canary requests, the probability of meeting at least one is 1 − (1 − p)ⁿ. At a 1% failure rate and 100 requests that works out to 1 − 0.99¹⁰⁰, near 63.4%, so a small canary can easily overlook rare but serious failures.

Base the canary design on risk as well as percentage. Seed it on purpose with risky tool paths, unusual languages, very long contexts, requests that sit on a permission boundary, queries returning nothing, and distinct groups of users. Shadow evaluation adds coverage with no side effects but can't expose every interaction effect of real actions.

Promotion and rollback triggers

Settle the triggers before the canary starts:

• an absolute policy violation means an immediate stop;
• a protected-data leak means a stop, the incident workflow and evidence preservation;
• a tool error or latency beyond its threshold means pause and diagnose;
• a statistically credible regression on the target means rollback;
• unexplained reward or judge drift means holding promotion;
• missing telemetry means failing closed, since the release can't be evaluated.

Nobody should invent acceptance criteria while staring at the dashboard. That turns governance into negotiation under deployment pressure.

Keep the rejected run on record

Audit records carry more weight when they include a candidate that failed, with the reason. Picture this case:

• the qlora-v4 run lifted tool validity three points;
• yet answers grounded in citations were five points less faithful once the serving quantization was applied;
• the gate turned it down;
• digging in, the team linked the drop to output projections that react badly to quantization;
• production kept running dpo-v1.

This proves the gate can stop a model that looks attractive on the surface. A registry holding only promoted runs can't show that the controls operate. A release manifest for the Reporting adapter might read like this:

release_id: inv22041-reporting-dpo-v1
base_sha256: "…"
reference_release: inv22041-sft-v3
adapter:
  method: lora
  rank: 16
  alpha: 32
dataset_digest: "sha256:…"
grader_version: sql-cage-12
policy_version: p-31
gates:
  tenant_binding: 1.0
  protected_leak_rate: 0.0
  tool_json_validity_min: 0.98
canary:
  mode: shadow
  duration_hours: 72
rollback_to: inv22041-sft-v3

The manifest names the release ID, the pinned base checksum, the reference release (the earlier SFT build) and the adapter details. It's small enough to review and complete enough to reproduce the claim.

End-to-end walkthrough: INV-22041

The defect report is specific: the base agent usually lands on the right answer, but malformed tool calls, poor recovery from empty SQL results and uncited causal language undermine it. Eight steps address that:

1. Locate the defect. Region mapping stays in governed memory, current sales data in the semantic layer and permissions in runtime policy. Training aims at tool grammar, recovery, citations and calibrated refusal.
2. Build the data plane. Compile 4,200 SFT examples plus 1,100 multi-step recovery traces, attach lineage, strip out draft finance documents and hold contamination_rate = 0 against the release suite.
3. Establish SFT. Train a LoRA adapter at rank 16 and alpha 32 on selected attention projections. Tool-parse success climbs from 0.71 to 0.94 while the general capability floors hold steady.
4. Add preference data. Raters pick cited, compact reports over fluent but unsupported alternatives. DPO runs with beta = 0.2 against the pinned SFT reference, reaching a 0.78 win rate with the leak rate still at zero.
5. Apply verifiable RL only where justified. For the Data Sufficiency stage, draw eight SQL and tool trajectories per prompt and grade them on tenant binding, validity, whether they run, what they cost and whether the data suffices. GRPO makes recovery better, and DAPO gets tried if identical-reward groups and long outputs weaken the signal.
6. Check architecture fit. Should the student be an MoE model whose token-level importance ratios destabilize RL, compare GSPO with the same grader and budget.
7. Distill when the economics demand it. Move observable tool behavior into a smaller local student using teacher outputs plus on-policy states, and repeat the leakage and tool suites on the exact quantized build.
8. Release the recipe. Pin each component, shadow for 72 hours, promote only once absolute floors and relative gains both pass, and leave one intentionally rejected run in the audit record.

At runtime the finished system still reads the current quarter, schema and permissions. Training improves its conduct around those facts without making it their owner.

Failure tree: where good training projects go wrong

The recurring pattern is that a proxy gets optimized while the enterprise contract is missing:

• Strong training accuracy may be masking a contaminated evaluation.
• Winning preference comparisons may come from longer answers or leaked citations.
• A climbing reward curve might reflect exploited graders.
• Students pick up the teacher's unsafe habits along with its skills.
• An adapter that looks flawless can break inside the quantized runtime that's really deployed.
• Rollback can fail if it restores weights but leaves the rest of the recipe in place.

The training objective proves only that optimization happened. Release evidence is the only thing that can show the enterprise system improved.

Proof of concept versus enterprise adaptation

Asking whether an idea moves a metric is fair at proof-of-concept scale. Enterprise scale adds proof obligations: lawful data, an independent metric, an improvement that survived integration, an attributable artifact and a reversible rollout.

So the most valuable post-training work often isn't a newer loss function. It's the engineering tying together data lineage, reward contracts, reproducibility, evaluation, registry identity, deployment, telemetry and rollback.

Twelve working rules

1. Keep changing facts in stores and stable behavior in adapters.
2. Pick the algorithm from the supervision and reward contract you actually hold.
3. Pin the reference model, grader, judge and policy version.
4. Use assistant-only loss for chat SFT, and verify the packing boundaries.
5. Prefer PEFT, and treat full fine-tuning as an exception that needs a written reason.
6. Training with QLoRA doesn't equal serving a quantized model.
7. DPO needs a pinned reference and beta, and reference-free methods still need baselines.
8. Unit-test every verifiable reward against adversarial trajectories.
9. Adopt DAPO or GSPO because a measured failure justifies it, and not because it's newer.
10. Distillation transfers behavior and failure modes alike, so evaluate the student independently.
11. Hold contamination at zero for release-defining suites.
12. Register the complete recipe, and rehearse rollback before promotion.

Contracts, not a ladder

Enterprise adaptation isn't a climb from SFT toward ever more fashionable acronyms. It's a chain of contracts.

The data contract states what the examples may teach, the objective names the optimized signal, and the environment lists the possible actions. Evaluation defines improvement, the registry records exactly what was built, the rollout fixes how much risk the organization will expose, and the rollback tells you whether the change is truly reversible.

With those contracts right, LoRA, DPO, GRPO, distillation and whatever post-training methods come next become useful engineering tools. Without them, even a beautiful reward curve is just a more expensive demo.