Soft-Label Distillation for a 0.8B SQL Agent: Why a Weaker Qwen Teacher Won

The short version

An earlier run took a 0.8B SQL agent from 1 solved task out of 220 to 46 by making it copy a bigger teacher one token at a time. This time only one thing changes. What happens if the student also sees part of the teacher's uncertainty, and not just the token it ended up choosing? Everything else stays put: the harness, the 220-task evaluation, the action schema, the 0.8B student and the LoRA fine-tuning setup. Only the learning signal moves.

The teacher's frozen traces were rescored. At each position the top 20 token probabilities were saved, along with a little leftover mass standing for everything else. Plain weighted hard-token training was then compared against top-k soft-label distillation. The best run solved 55 of 220 tasks, against 46 for the hard-token student and 49 for a combined hard-token run.

The headline number isn't the most interesting part. Soft labels made things worse in two of the eight single-source runs. The best teacher wasn't the strongest one, either. Qwen, a weaker model from the same family at 96/220, matched its own traces much more closely than the stronger GPT model (115/220) did, so its probabilities gave the student a cleaner lesson. The real win came from mixing the two sources.

One caveat belongs up front. GPT's own token probabilities were never available. The GPT traces here were scored by Qwen, which makes them a proxy and not true GPT logit distillation.

What was being tested

The broader goal is making small models better at narrow tasks. The agent is the same SQL repair agent as before. It receives a user issue plus some buggy SQL, can inspect a SQLite database, run probe queries, read the results and submit corrected SQL. Scoring comes from hidden deterministic tests, with no LLM judge.

The earlier hard-token run had already shown that a small model can operate this loop. The untuned base model mostly repeated actions and never submitted anything. After hard-token SFT it inspected, queried and submitted far more often.

Training on hard tokens is a crude instrument. The student is told to reproduce whatever token the teacher wrote, with no hint of how sure the teacher was, what else it considered, or whether the second-best option was nearly as good. For SQL that missing information can matter in table and column names, predicates, joins, literal values and the syntax of structured actions.

The question, then, was whether top-k soft-label distillation improves the same 0.8B agent when traces and benchmark are held fixed.

Two starting points were compared. One was the base unsloth/Qwen3.5-0.8B model. The other was the Qwen3.5-0.8B hard-token adapter, trained earlier on successful trajectories from Qwen3.5-35B-A3B 8-bit. Put differently: should soft-label training start from scratch, or carry on from the hard-token student?

The SQL agent task

The setup matches the earlier hard-token run. Each task has a user issue, a buggy query and a SQLite database. The model picks among structured actions: inspect the schema, run SQL, or submit final SQL. A task counts as solved only if the submitted SQL passes the hidden tests.

A typical small but real task: on the chinook database, the user wants the latest track_id and then wants to use it to filter rows of the track table. The buggy query is:

WITH vars AS (SELECT COUNT(*) AS vars_id FROM track)
SELECT * FROM track WHERE track_id = vars_id

The flaw has nothing to do with formatting. COUNT(*) isn't the latest id, and a repair along the lines of MAX(track_id) is needed. The model has to work that out inside a tool loop: perhaps inspect the schema, run a probe, read the observation, and finally submit SQL in the exact structured format the harness accepts.

BAML, short for Boundary Markup Language, still wraps the prompt as the structured-output layer. It pins down the shape of a decision and assembles the request sent to the model. Free-form teacher prose is never part of the student's training data; what it learns are canonical JSON decisions made of a brief draft and an executable output action:

{"draft":"Need schema first.","output":{"action":"inspect_schema"}}
{"draft":"Test a candidate query.","output":{"action":"run_sql_query","sql":"SELECT ..."}}
{"draft":"Submit corrected SQL.","output":{"action":"submit_sql","sql":["SELECT ..."]}}

Every evaluation reused the settings of the earlier run, so the training signal was the only thing that changed. Two settings deserve a word. The 8192 sequence cap is the total budget for prompt and generation together: system prompt, task, earlier actions, schema, query results, structured-output instructions and the generation reserve all share it. Maximum generated tokens is listed separately because it limits a single model call. The student can read a long state but must still emit one compact action per turn. Results are reported through terminal outcomes and behaviour metrics such as submissions, wrong submissions, SQL errors, repeated-action stops and max-turn stops.

The best students aren't failing mainly on JSON. They reach submit_sql without trouble. The hard part is SQL judgment and recovering when a probe goes badly.

The training idea

Hard-token distillation gives the student a single target token per position. Soft-label distillation supplies a small probability distribution instead. Suppose the teacher's target contains a table token as in ... FROM users .... Hard training says the next token is users. Soft-label training can also say how much probability the teacher gave to a couple of near alternatives. The chosen token still counts, but the student additionally sees that some alternatives sat closer than others.

Storing full-vocabulary probabilities is costly for a large language model. A vocabulary may hold hundreds of thousands of tokens, and each teacher decision spans many assistant target positions. The practical compromise was a sparse version: the top 20 teacher probabilities per assistant target position, plus the residual tail mass for every other token.

Why keep the tail? Say the saved top 20 probabilities add up to 0.996. The remaining 0.004 still belongs to the teacher's distribution, and squeezing the top 20 back up to 1.0 would give the student an overconfident target. So the target is deliberately built to remember that some mass lives outside what was saved.

The approach is still off-policy. The student learns from traces collected earlier and never acts while scoring takes place.

A short code walkthrough

The simplest way to follow the method is to track one row through the pipeline. No new agent environment was built for soft labels. The same successful teacher decisions from the hard-token trajectory run were reused, rendered with the same BAML prompt shape, and then enriched with probability information on the assistant target tokens.

Here is a hard-token supervised fine-tuning (SFT) row. SFT is training on input-output pairs where only the answer tokens contribute to the loss:

row = {
    "messages": conversation_before_teacher_decision + [
        {"role": "assistant", "content": canonical_decision_json(teacher_decision)}
    ],
    "teacher_draft": teacher_decision.draft,
    "teacher_action": teacher_decision.output,
}

A soft-label row keeps the same canonical messages and teacher metadata and gains teacher-forced token scores at the assistant target positions. "Teacher forcing" here means handing the known target text to the probability model and asking, token by token, how likely that text was given its context. Log probabilities are stored rather than raw ones, which are safer numerically to write out and train on:

row = {
    "messages": [..., {"role": "assistant", "content": target_decision_json}],
    "distillation": {
        "probability_model": "Qwen3.5-35B-A3B 8-bit",
        "top_k": 20,
        "row_weight": 1.18,
        "token_scores": [
            {
                "position": 1661,              # full-sequence token position
                "target_token_id": 412,
                "target_logprob": -0.21,
                "top_token_ids": [412, 879, 91],
                "top_logprobs": [-0.21, -2.81, -3.51],
                "top_mass": 0.90,
                "tail_mass": 0.10,
            }
        ],
    },
}

The scoring pass has a single job: take a frozen row and stamp token probabilities on it. The Qwen3.5-35B-A3B 8-bit scorer is never asked to write a new decision. The prompt is rendered without the assistant answer, the full sequence is rendered with it, and the target portion is scored one token at a time:

prompt = chat_template(messages[:-1], add_generation_prompt=True)
full = chat_template(messages, add_generation_prompt=False)
prompt_ids = tokenize(prompt)
full_ids = tokenize(full)
target_ids = full_ids[len(prompt_ids):]
prefill_teacher_cache(prompt_ids[:-1])
driver_ids = [prompt_ids[-1]] + target_ids[:-1]
for i, target_id in enumerate(target_ids):
    logits = qwen_35b(driver_ids[i]).logits
    logprobs = log_softmax(logits)
    top_ids, top_logprobs = top_k(logprobs, k=20)
    save_token_score(
        position=len(prompt_ids) + i,
        target_token_id=target_id,
        target_logprob=logprobs[target_id],
        top_token_ids=top_ids,
        top_logprobs=top_logprobs,
        tail_mass=1.0 - sum(exp(top_logprobs)),
    )

Assistant-only masking is unchanged from the hard-token run. Prompt tokens are context, and assistant target tokens are supervision. That matters more for an agent than for plain question answering, because the prompt carries schema observations, earlier tool output and SQL results. Those tokens are there to be conditioned on, never predicted:

labels = full_ids.copy()
labels[: len(prompt_ids)] = -100
hard_ce = cross_entropy(student_logits, labels)

Training also needs more tensors than ordinary SFT. A batch carries input_ids, attention_mask and masked labels as usual, and on top of those it also holds a weight for each position, the ids of the top-k tokens with their log probabilities, a mask over those k slots, and the leftover tail probabilities. Padding puts neutral values in inactive slots, so the loss ignores them without fuss:

batch = {
    "input_ids": pad(input_ids, pad_id),
    "attention_mask": pad(attention_mask, 0),
    "labels": pad(labels, -100),
    "position_weights": pad(position_weights, 0.0),
    "topk_token_ids": pad(topk_token_ids, [0] * top_k),
    "topk_logprobs": pad(topk_logprobs, [0.0] * top_k),
    "topk_mask": pad(topk_mask, [False] * top_k),
    "tail_probs": pad(tail_probs, 0.0),
}

The control condition is weighted hard-token fine-tuning. One chosen token is still the target, but assistant-token cross-entropy gets multiplied by a confidence weight taken from the scorer's log probability for that target. An easy row for the scorer can weigh more, and an unlikely one can weigh less:

z = (mean_target_logprob - dataset_mean_logprob) / dataset_logprob_std
row_weight = clip(2 / (1 + exp(-z)), 0.25, 1.75)
loss = row_weight * hard_ce

Clipping to [0.25, 1.75] prevents any single odd row from taking over a batch, while confident rows can still outweigh shaky ones by a wide margin.

On the soft-label side, a probability-matching term is added to the loss. What follows is a sketch of its shape and not the exact production code:

teacher_top_probs = exp(top_logprobs) * topk_mask
teacher_top_probs *= (1.0 - tail_mass) / sum(teacher_top_probs)
student_logprobs = log_softmax(student_logits_at_target_positions)
student_top_logprobs = gather(student_logprobs, topk_token_ids)
student_tail_prob = 1.0 - sum(exp(student_top_logprobs))
topk_kl = sum(teacher_top_probs * (log(teacher_top_probs) - student_top_logprobs))
tail_kl = tail_mass * (log(tail_mass) - log(student_tail_prob))
loss = hard_ce + topk_kl + tail_kl

Information flows from teacher to student: the teacher's distribution is what the student aims at, and it takes a penalty when it assigns too little probability to that mass. In the real trainer, cross-entropy and distillation are both computed over shifted assistant target positions only.

Actual student training is still LoRA fine-tuning. LoRA, low-rank adaptation, keeps the base model mostly frozen and attaches small trainable adapter matrices to chosen attention and MLP modules. That made it cheap to compare many runs without fully fine-tuning the 0.8B model each time:

student = load_student("unsloth/Qwen3.5-0.8B", max_seq_length=4096)
student = add_lora_adapters(
    student,
    rank=32,
    alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)
trainer_class = make_trainer_class(Trainer, soft_label_args)
trainer = trainer_class(
    model=student,
    train_dataset=scored_train_examples,
    eval_dataset=scored_validation_examples,
    data_collator=distillation_collator,
    batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=5e-5,
    epochs=3,
)
trainer.train()
adapter = save_lora_adapter(student)

The real implementation mirrors this: make_trainer_class hands back a Hugging Face Trainer subclass that applies the saved sparse top-k plus residual-tail loss. Note how that differs from TRL's online DistillationTrainer, in which the teacher scores inside the trainer. All three soft-label weights were left at 1.0: hard cross-entropy, distillation and temperature. Temperature stayed plain on purpose, because the sparse artifacts were written at 1.0 and a temperature-scaling study was not the aim.

Evaluation at the end is a genuine rollout rather than teacher forcing. The trained adapter returns to the SQL-agent harness and has to work through the 220 held-out tasks, acting one turn at a time:

for task in held_out_eval_tasks:
    state = start_sql_agent_task(task)
    for turn in range(8):
        messages = render_baml_messages(state)
        decision = model_with_adapter.generate_action(
            messages,
            max_new_tokens=512,
            temperature=0.0,
        )
        state = execute_action_and_append_observation(state, decision)
        if state.solved or state.stopped:
            break
    results.append(score_final_submission(state))

Many offline distillation ideas break down at this last step. Teacher forcing tests whether the student can predict what the teacher would do next in a context the teacher shaped. A rollout tests whether the student can live with its own earlier choices, pick useful probes, avoid repetition and submit correct SQL. Only the second number matters here.

How the experiment runs

It would be easy to read this as on-policy distillation, and it is not. Nothing the student generates is ever scored. It only learns, afterwards, from traces the teacher had already produced.

The data comes from a scoring pass over frozen successful traces. The teacher's actions exist already, and the probability model only asks how likely each chosen token looked given the context. It never writes a fresh decision. The two scored datasets were kept apart (GPT traces scored by Qwen, Qwen traces scored by Qwen) and each was filtered by the length of the whole rendered sequence, since schema observations and earlier tool output are part of what the student must absorb, rather than only the closing answer. Boundary: the GPT traces count as Qwen-proxy-scored and were not scored with GPT's own probabilities, and the student never acts during scoring.

Three training variants were applied to the scored data:

• Weighted hard-token fine-tuning (the control). One chosen token per position, with each row weighted by the scorer's confidence. It is SFT with a confidence knob. Boundary: this isn't soft-label distillation, because the student sees the weight and never the top-k distribution.
• Top-k soft-label distillation (the method under test). The same hard target plus a second signal that asks the student to spread probability over the top-k alternatives the scorer saw, reserving leftover mass for everything else. Boundary: it is off-policy and teacher-forced over frozen traces, and the teacher never scores the student's own samples.
• A combined teacher-source mix. GPT-proxy and Qwen-self rows are joined, shuffled, and used to train the base student both ways. Boundary: it mixes data across teachers instead of merging their distributions into one, and each row keeps the scorer that produced it.

Mixing sources gives the cleanest result, and the least puzzling one: more varied successful traces plus a richer per-token target handed the small base model more to learn from than either source did alone.

Dataset and filtering

Teacher traces weren't regenerated, on purpose. They were frozen from the hard-token run so the comparison stays about the supervision signal, not about a new round of data generation.

In that earlier generation pass, both teachers ran over the same 879 training tasks. GPT 5.5 medium got 446 of the 879 right, which yielded 1046 SFT rows. Qwen3.5-35B-A3B 8-bit got 394 right and yielded 1232 rows, because its successful traces ran longer. On the 220-task evaluation GPT 5.5 medium was stronger, 115 against 96, so the Qwen path here is a same-family test and not the strongest-teacher path.

The proxy caveat carries weight. Honest GPT 5.5 medium soft labels would need that model's own token probabilities, and they weren't available. So the GPT-trace experiment poses a narrower question: does a same-family Qwen scorer add anything useful to GPT's chosen actions?

For every row, the BAML prompt the harness really uses was rendered. The known assistant target was then teacher-forced through Qwen3.5-35B-A3B 8-bit and token scores were stored only for assistant target positions.

Before any training, two things about the scored data needed checking: what share of it fit inside a 4096-token budget, and how closely the Qwen scorer agreed with each trace source. Lower negative log likelihood is better, because it means the probability model finds the target actions easier to explain. That is why Qwen self-scored data differs so much from GPT proxy-scored data: Qwen was far more confident in its own traces than in GPT 5.5 medium's. Mean target negative log likelihood came to 0.297 for Qwen self-scored rows and 0.876 for GPT proxy-scored rows. The target token sat inside the stored top 20 at every Qwen self-scored position (100.00%) and at 98.56% of GPT proxy-scored positions.

The merged set started at 2278 rows. After the 4096-token filter 2253 remained, divided into 2141 for training and 112 for validation. A fixed seed shuffled the rows before the split, so the validation set wouldn't reflect the order of the sources.

Training setup

Eight runs formed the main grid, a full crossing of three binary choices: which trace source (GPT proxy or Qwen self), which method (weighted hard or soft label) and which student start (base or hard-token warm start). Two more runs, both from the base model and both on the combined data, one weighted hard-token and one soft-label, made ten trained students in total. Combined warm-start runs were left out. The first eight showed that warm starts depended heavily on the trace source, and the combined runs were meant to answer something simpler: from the base model, does mixing teachers help?

Training hyperparameters followed the hard-token trajectory baseline. The adapters used LoRA with rank 32 and alpha 32 across the attention projections (q_proj, k_proj, v_proj, o_proj) and the MLP projections (gate_proj, up_proj, down_proj). Other settings: a 5e-5 learning rate, a per-device batch of 1 with 8 accumulation steps, three epochs, bf16 on an NVIDIA GPU, Unsloth, and FlashAttention 2 whenever available. Remaining settings were a 5% validation split, packing off, seed 42 and a 4096-token ceiling on full training sequences.

For soft labels the temperature was 1.0, with weights of 1.0 on both hard cross-entropy and distillation. The code keeps the stored top-k mass alongside the residual tail mass, a plain numerical fix for probabilities that were serialised sparsely.

Run sizes weren't identical. GPT proxy-scored runs trained on 990 rows, Qwen self-scored ones on 1151 and combined ones on 2141. At batch size 1 with 8 accumulation steps, three epochs came to 372, 432 and 804 optimizer updates respectively. In the run labels, Base is the untuned unsloth/Qwen3.5-0.8B, GPT proxy means traces from GPT 5.5 medium that Qwen3.5-35B-A3B 8-bit rated, Qwen self means that Qwen model's traces scored by itself, and Combined is both scored sets joined before training.

Be careful with the validation losses. They mean something inside a single objective, but weighted-hard and soft-label runs can't be ranked against each other because their losses include different terms. The lowest validation loss belonged to the base Qwen self-scored weighted-hard run, which solved only 40/220, while the best rollout came from the combined soft-label run despite its higher loss.

Preparation and training landed on different hardware. A high-memory machine was convenient for loading and scoring Qwen3.5-35B-A3B 8-bit locally, though exact teacher-forced scoring crawled: roughly 60 minutes for the GPT traces and 41 for the Qwen traces. Students trained on the NVIDIA GPU. Those single-source runs with recorded durations lasted about 23 to 30 minutes apiece, and the larger combined runs took around 52 minutes (weighted hard) and 55 (soft labels).

Results, organised by question

A first chart tried to show everything at once and confused two separate questions: what happens when training starts from the base model, and what happens when it continues from the hard-token trajectory model. The results were therefore split by research question.

For scale, the teachers and baselines on the fixed 220-task evaluation were: no fine-tuning 1/220, the hard-token student 46/220, Qwen as a teacher 96/220 and GPT as a teacher 115/220. The aim of the top-k soft-label experiment was never to beat the teachers. It was to see whether a richer supervision signal could push the 0.8B student further under the same harness.

Starting from the base model. All trained runs beat the untuned 1/220. GPT-only and Qwen-only training ended up near each other, with 41 to 43 tasks solved depending on the method. Combining the two sources made the larger difference: the combined weighted hard-token run got 49/220 and the combined top-k soft-label run got 55/220, roughly a quarter of the tasks. Modest in absolute terms and far from the best teacher's 115/220, but the interest lies in how far a 0.8B model climbs from nearly zero, not in matching the teacher. For a base-model start, the conclusion is to draw on both teachers, with soft labels ahead of weighted hard tokens on that merged data.

Continuing from the hard-token student. Here the picture is less clear. The checkpoint already solved 46/220. More training on GPT traces with weighted hard tokens nudged it to 48/220, whereas soft labels on those same traces slid back to 44/220. On Qwen self-scored traces the pattern reversed: weighted hard tokens fell to 38/220, and soft labels rose to 49/220.

The unexpected finding: soft labels helped only when the scorer and the trace were in agreement. Qwen3.5-35B-A3B 8-bit was the weaker teacher at 96/220 versus GPT's 115/220, yet its probabilities fit its own traces, and soft labels worked there (49/220) while hurting on the proxy-scored GPT traces (44/220). Agreement outweighed raw teacher strength.

What changed in the agent loop. The base model almost never submitted. Hard-token trajectory SFT taught it to enter the loop and submit, but many of the submissions were wrong. The combined soft-label run lifted success again and is the best run overall. Even so, it submitted 171 tasks and solved 55, meaning 116 submitted queries failed the hidden tests, and it still had 39 repeated-action stops.

That is the honest shape of the result. The model isn't mainly failing because it can't produce valid BAML actions. It fails because it still makes poor SQL decisions and sometimes gets stuck in the loop.

The most reliable behaviour signal was turn count. The combined soft-label run averaged 3.39 turns against 3.48 for the hard-token warm start, close enough that extra steps can't explain the improvement. A slightly better action mix does: more correct submissions than the warm start, fewer wrong ones than the combined weighted-hard run, along with fewer SQL errors and fewer stops at the turn cap. Generated-token counts per evaluation call weren't logged, so no output-token figures are offered. The numbers that are safe to report come from training: how many prompt and target tokens there were, the full sequence length and the top-k coverage.

A note on reading the tables behind the charts: "submitted" covers both correct and wrong submissions, and "wrong submits" equals submitted minus success. The "SQL-error tasks" column is non-exclusive. A task lands in it if any one of its SQL probes threw an error mid-loop, so it may coincide with any final result.

Failure analysis

Judged purely on the combined base-model run, the outcome would look too neat: mixed teachers plus soft labels, case closed. The split views are less tidy. The surprising headline is that a weaker teacher taught better than a stronger one, since its probabilities were in line with its own traces. Soft labels gained the most on same-family Qwen paths and plainly so when the sources were combined. They gained less on GPT traces that Qwen had only scored by proxy, and they hurt the GPT warm-start cell.

This is plausible: Qwen3.5-35B-A3B 8-bit probabilities describe Qwen3.5-35B-A3B 8-bit actions well, so the 0.8B Qwen student gets a cleaner signal to imitate.

Rollout keeps the last word. Under teacher forcing the question is whether the student can guess the teacher's next action when the context looks like the teacher's. The harness asks something harder: can it recover from its own earlier moves, pick sensible probes, stay out of loops and finish with correct SQL? The two are linked but far from the same.

Validation loss tells the same story from a different side. At 55/220 the combined soft-label run had a validation loss of 0.5684, higher than the 0.3478 of the combined hard-token run, which solved 49. The lowest loss anywhere, 0.2544, belonged to the base Qwen self-scored hard-token run, and it solved just 40/220. Low teacher-forced loss did not equal a better agent.

The combined-teacher result teaches the most. GPT traces pushed the student to submit more often. Qwen traces were easier to score and from the same family, yet they brought more looping. The mixed soft-label run didn't simply take the best of each, and repeat stops persisted. Even so, its success count was the highest, and compared with the mixed hard-token run it hit fewer SQL execution errors and ended fewer episodes at the turn limit.

The weighted-hard control had a lesson of its own. Weighting rows by scorer confidence is cheaper than fitting sparse probability distributions, and on GPT proxy-scored warm-start data it produced the best cell. What it cannot do is tell the student about neighbouring alternatives at each position; it says only "copy this token harder or softer". Top-k soft labels suit cases where the scorer matches the trace source and the mix of data is wide enough that the student doesn't overfit to a single teacher's style.

Hardware lessons

Two kinds of machine were needed. Scoring meant keeping Qwen3.5-35B-A3B 8-bit in memory and running it across frozen targets, so a high-RAM Mac with 128 GB of unified memory, which can hold the quantised teacher resident, made sense. It wasn't fast, but it allowed all the preparation to happen locally before any paid GPU time.

Training wanted the opposite. LoRA fine-tuning is matrix-heavy, with forward and backward passes, attention kernels and optimizer steps, and the NVIDIA GPU was much faster despite less total memory, since those tensors stayed in VRAM and ran on CUDA kernels. The recurring lesson is that capacity and throughput are separate problems. Big system RAM lets you fit and prepare things, while GPU VRAM plus CUDA lets you train them quickly. The workflow settled into doing everything possible on the Mac and starting the remote GPU only once data and scripts were ready.

Lessons

Soft-label distillation merits a try for agent fine-tuning, provided it is judged by rollouts. Token-level uncertainty carries more information than hard tokens, but it remains local supervision and won't by itself teach a model to recover from a bad tool call, avoid loops or probe better when new observations arrive.

Proxy soft labels should be labelled honestly. The GPT rows here aren't GPT's own probabilities but Qwen-scored ones. That's still a useful experiment, just not true GPT logit distillation.

A teacher from the same family may beat a stronger one. How well the scorer matched the trace counted for more than its evaluation score.

Sparse top-k with a tail is a workable compromise. Saving full-vocabulary logits would be more complete but is pricey to store, move and train on. Twenty tokens plus a residual tail preserved enough that the target token stayed visible nearly everywhere.

Mixing teachers beat picking one winner. The best result combined GPT 5.5 medium proxy-scored rows with Qwen self-scored rows and trained the base model with soft labels.

Where it leaves things

So, does it help? Yes, under conditions. In the best setup, top-k soft-label distillation lifted the 0.8B SQL agent: GPT proxy-scored rows combined with Qwen self-scored rows, fed to the base unsloth/Qwen3.5-0.8B student, solved 55/220, clearly ahead of the off-policy hard-token trajectory student (46/220) and the combined hard-token run (49/220).

Don't read this as "soft labels always win". Teacher source, student initialisation, probability alignment and rollout behaviour all mattered, and the teacher that came out ahead wasn't the one with the best evaluation score. What still goes wrong is agentic: wrong SQL, repeated actions and imperfect recovery inside the loop.

This is exactly why the benchmark is worth keeping. A student can look fine under teacher forcing and still stumble once it must act. Any follow-up should hold the harness constant and check whether a new training idea improves the actual loop and not just the offline loss.