Serving a LoRA Fine-Tune Locally: Verify, Fuse and Avoid Silent Failures

Finishing a LoRA training run leaves you with a small adapter file, around 11 MB in this case, and not much else. A file is not a result: until the fine-tuned model is scored against the same baseline and served where an application can call it, you only have hope. This guide takes a support-ticket triage adapter for a 2B-parameter model, measures it, fuses it into standalone weights, serves it behind a local OpenAI-compatible endpoint, and walks through the failure modes that return plausible, well-formed and wrong answers without a single error message.

Everything shown runs from the finetune-demo repository, which includes the trained adapter, so you can follow along without training anything yourself. The headline: on this task the model went from zero fully valid answers out of 40 to 40 out of 40.

Measuring the adapter against the baseline

The only fair measurement changes exactly one variable. The evaluation uses the same script, the same 40 held-out tickets and the same temperature as the baseline run on the untrained model; the only addition is the --adapter flag pointing at the trained weights. --no-think disables the model's reasoning mode so it answers directly.

python evaluate.py - model mlx-community/Qwen3.5–2B-MLX-4bit \
--adapter adapters/triage-2b --limit 40 --no-think

===== mlx-community/Qwen3.5–2B-MLX-4bit (adapter: adapters/triage-2b) =====
examples : 40
usable : 40/40 (100%) returned parseable JSON
fully valid : 40/40 (100%) <- the headline
median latency: 0.33s
errors by rule:

The errors by rule section is empty, which is the entire point: every one of the 40 responses satisfied every validation rule. Median latency was 0.33 seconds per ticket.

Laid side by side with the baseline, the change is stark. The untrained model already produced parseable JSON every time, but it never used the required vocabulary for category, priority or tags:

|                         |  Before   | After     |
|-------------------------|-----------|-----------|
| Returned parseable JSON |   40/40   | 40/40     |
| **Fully valid**         |  **0/40** | **40/40** |
| `category` errors       |     40    | 0         |
| `priority` errors       |     40    | 0         |
| `tags` errors           |     40    | 0         |
| `needs_human` errors    |      8    | 0         |

The same ticket used to demonstrate the baseline shows why. Before training, the model invented labels such as "IT Support" and Title-Case tags; afterwards it used the house schema's lower-case values:

TICKET : The password reset email never arrives, I have checked spam.

BEFORE : {"category": "IT Support", "priority": "High", "needs_human": true,
          "tags": ["Password Reset","Email Delivery","Account Access","Spam Filter"]}

AFTER  : {"category": "account", "priority": "medium", "needs_human": true,
          "tags": ["password", "email_change"]}

There is a second gain in that example. The base model used 63 completion tokens for its answer, the fine-tuned one 29, less than half. Output tokens set both response time and the inference bill on a busy endpoint, so halving them is a material saving, not a rounding error.

Trying it on your own text

Because the adapter ships with the repository, the try_it.py script works right after cloning. Passing --compare loads both the base model and the adapted one, so you can see the difference on a ticket you wrote:

.venv/bin/python try_it.py \
--compare "I was charged twice for my Pro plan and nobody has replied in a week"

TICKET       "I was charged twice for my Pro plan and nobody has replied in a week"

before       { "category": "Billing & Support", "priority": "High", "needs_human": true,
                 "tags": ["Duplicate Charge","Account Inquiry","Support Ticket","Pro Plan"] }
               INVALID -> category, priority, tags   (0.42s)

after        {"category":"billing","priority":"medium","needs_human":true,
                "tags":["double_charge","email_change"]}
               VALID   (0.24s)

The base answer fails validation on three fields; the adapted answer passes and is also faster. Drop --compare to get only the fine-tuned answer, or leave out the ticket text to get an interactive prompt.

Three ways to run the model

You can keep the adapter separate, fuse it into the base weights, or convert to another format. Fusing is the most robust for serving.

Fusing the adapter

LoRA represents the weight update as a low-rank product BA that is added to the frozen weights W on every forward pass. Fusing performs the W + BA addition once and writes ordinary weights, giving you a single self-contained model directory:

python -m mlx_lm fuse \
  --model mlx-community/Qwen3.5-2B-MLX-4bit \
  --adapter-path adapters/triage-2b \
  --save-path fused/triage-2b

This took 3.6 seconds and produced 1.0 GB of output. The next step is not optional: score the fused model before trusting it.

fused/triage-2b   fully valid: 40/40 (100%)   median latency 0.26s
adapter           fully valid: 40/40 (100%)   median latency 0.33s

Quality is identical, and the fused model is measurably faster, because the extra matrix multiplication per layer has disappeared. The reason to re-score is that fusing is arithmetic, and incorrect arithmetic fails silently. A broken fuse still produces a directory of reasonable-looking files that then generate confident nonsense. Only a score distinguishes the two.

Serving it

mlx_lm server exposes the fused model over HTTP. The --chat-template-args option switches off thinking at the server level, which matters for reasons covered below:

python -m mlx_lm server --model fused/triage-2b --port 8082 \
       --chat-template-args '{"enable_thinking":false}'

A plain curl request against the chat completions endpoint confirms the model answers in the trained format. Temperature is zero for deterministic output, and the system prompt is the one used during training:

curl -s -X POST http://127.0.0.1:8082/v1/chat/completions \
 -H 'Content-Type: application/json' -d '{
 "messages":[
  {"role":"system","content":"You are a support triage engine. Reply with one JSON object and nothing else, with keys: category, priority, needs_human, tags."},
  {"role":"user","content":"Production is down for all our users. The app crashes every time I open the dashboard screen."}],
 "max_tokens":120,"temperature":0}'


{"category": "bug", "priority": "urgent", "needs_human": false,
 "tags": ["crash", "desktop"]}

The reply used 29 completion tokens. Because the endpoint is OpenAI-compatible, existing code written against the OpenAI API can use it by changing only the base URL.

Calling the endpoint from application code

The integration fits in one function using only the Python standard library. The version in the repository's client_example.py imports the system prompt and the validation helpers from a shared schema module, posts the request, and refuses to return anything it cannot validate:

import json, urllib.request
from schema import SYSTEM_PROMPT, validate, extract_json

ENDPOINT = "http://127.0.0.1:8082/v1/chat/completions"

def triage(ticket_text, timeout=60):
    payload = {
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},   # MUST match training
            {"role": "user",   "content": ticket_text},
        ],
        "max_tokens": 160, "temperature": 0,
    }
    req = urllib.request.Request(ENDPOINT, data=json.dumps(payload).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        body = json.load(r)

    msg = body["choices"][0]["message"]
    content = msg.get("content")
    if not content:                                  # thinking left no answer
        raise RuntimeError(f"no content; finish_reason={body['choices'][0]['finish_reason']}")

    record = extract_json(content)
    errs = validate(record) if record is not None else ["unparseable"]
    if errs:                                         # never trust it blindly
        raise ValueError(f"invalid record: {errs} -> {content!r}")
    return record

Run against two tickets, it returns clean dictionaries:

I was charged twice for my Pro subscription this month.
  -> {'category': 'billing', 'priority': 'medium', 'needs_human': True,
      'tags': ['double_charge', 'invoice']}

Production is down for all our users, the dashboard crashes on load.
  -> {'category': 'bug', 'priority': 'urgent', 'needs_human': False,
      'tags': ['crash', 'desktop']}

Three details in that function are there on purpose, and each one guards against a failure described in the next section:

• SYSTEM_PROMPT comes from an import rather than a copy. Even a one-character difference from training moves the model off-distribution.
• The empty-content check. If the model spends its entire allowance on reasoning, there is no answer to parse.
• validate() runs on every response. A fine-tuned model has a strong tendency, not a guarantee. A perfect score on a test set says nothing certain about the next request, so decide in code what happens when a record fails.

Three failures that never raise an error

None of the following throws. Each returns a confident, well-structured, wrong answer.

The adapter flag that is silently ignored

The obvious shortcut is to skip fusing and hand the adapter straight to the server:

python -m mlx_lm server --model <base> --adapter-path adapters/triage-2b

With mlx-lm 0.31.3, the version used here, this served the base model. There was no warning, no log line and no error. The endpoint started normally and replied with "category": "Production", "priority": "Critical" and a set of four Title-Case tags: the untrained model's behaviour, unchanged. Without a baseline number to compare against, the natural conclusion would have been that the fine-tune failed. Later versions may behave differently, so verify rather than assume.

A quick way to catch this takes seconds: send a request whose correct answer you already know. A reply in your own label vocabulary means the adapter is active; a reply that resembles the base model means it is not. The fused route, scored above, avoids the question entirely.

Reasoning that consumes the whole budget

Many recent small models reason before answering. Request JSON with a 120-token limit while thinking is enabled, and the response can look like this:

{
   "choices":
   [
    {
      "finish_reason":"length",
      "message":{
        "role": "assistant",
        "reasoning":"Thinking Process:\n\n1. **Analyze the Request:** ..."
      }
    }
   ]
}

There is no content field. Every token went to reasoning, generation stopped with finish_reason: "length" mid-thought, and a client that reads response.choices[0].message.content either hits a KeyError or, worse, gets an empty string that it treats as a legitimate empty answer.

Disable thinking on the server with --chat-template-args '{"enable_thinking":false}', or per request with "chat_template_kwargs": {"enable_thinking": false}. With thinking off, the same request completed in 29 tokens.

A system prompt that differs from training

Training taught the adapter to answer under exactly one system prompt. Change that prompt and the request falls outside what it saw, and most of the learned behaviour vanishes. Here is the same fine-tuned model given a generic prompt asking a helpful assistant to categorise the ticket:

This is a **Critical Production Incident** (or a **Major Service Level Incident**).

Here is the breakdown of why this categorization applies:

*   **Severity Level: Critical / P0**
    *   **Impact:** Total system outage affecting all users.

The result is a Markdown essay with no JSON at all. The model is not broken; it was asked a question it was never trained on. Keep a single definition of the prompt, shared by the data generator and the client, and import it everywhere.

Two more traps: the model list and your own harness

GET /v1/models lists every model in the local cache, not the one currently loaded. Think of it as a cache listing rather than a health probe: it can tell you the server is up, but not which weights are answering.

Also check the evaluation harness before blaming the weights. In this project the evaluator decided whether to disable thinking by looking for "qwen" in the model name. That worked for mlx-community/Qwen3.5-2B-MLX-4bit, but the fused copy lives at fused/triage-2b, so thinking stayed on without anyone noticing and the fused model scored 82% rather than 100%. The weights were fine; the evaluator was at fault. When a score drops unexpectedly, suspect the harness first, and never branch behaviour on a file name.

What the 40/40 does not prove

The perfect score is real, but be precise about its scope: it covers held-out tickets created by the very generator that produced the training set. The model does generalise, but only to new synthetic examples of that kind.

A handful of realistic, messy tickets tells a different story. Six were tried. Four passed structural validation, and several of those were still wrong with full confidence:

• An all-caps complaint that orders could not ship and everything was broken landed in account instead of bug.
• A thank-you note praising a fix to the dashboard was pushed into feature_request, since the schema offers no "not a ticket" option and the model has to choose one.
• A GDPR deletion request became how_to with needs_human: false, routing a legal deadline away from a person.

The last one is a data defect, not a model defect. In the generated dataset, needs_human is completely determined by category:

account {True: 125}   billing {True: 137}
bug {False: 153}      how_to {False: 115}      feature_request {False: 110}

The model therefore learned a five-row lookup table rather than a judgment, and no amount of training can fix a label that was never independent. You only discover this by testing off-distribution, so treat a held-out score as the minimum you can claim rather than the maximum. For production use, label a few hundred real tickets, let needs_human vary independently of category, and introduce a "no action" label.

Beyond support tickets

Nothing in this pipeline is specific to tickets. It fits anywhere you have unstructured text and a fixed label set:

• CVs into seniority, years of experience and skill tags.
• Invoices into vendor, currency and line-item categories.
• Log lines into service, severity and incident type.
• Reviews into sentiment, feature mentioned and defect type.

Only two files need to change: schema.py, which holds your allowed values, prompt and validate() function, and make_data.py, which produces your examples. Every command shown here then works unchanged.

Before you fine-tune the next model

Try constrained decoding first. GBNF grammars in llama.cpp or libraries such as xgrammar force generated output to match a schema, which makes structurally broken output impossible whether or not the model was fine-tuned. Applying a grammar alone would have pushed schema validity to 100% here with no training. Fine-tuning still earned its place: a grammar can enforce shape but not meaning, and training is what taught the model the right category, while also cutting the token count in half. But if malformed JSON is your only problem, reach for a grammar before a training run.

Count cost per request, not per training run. The training run cost roughly five minutes, a single time. Token spend recurs with every call as long as the service exists, so dropping from 63 to 29 tokens is the saving that keeps growing. If you are weighing this against a hosted API, the analysis in fine-tune or call the API walks through the numbers for a similar pipeline.

Treat GGUF as the fragile third path. A GGUF conversion makes the model portable to llama.cpp or Ollama, yet the tooling may finish without complaint and leave you with weights that emit garbage. Produce a sample completion after each conversion step; the presence of a GGUF file proves nothing about whether it runs correctly.

Key takeaways

• The number that gives every later result meaning is the baseline. Measure before training, and re-measure after every transformation such as fusing or conversion.
• Fused weights were as accurate as the adapter and faster to serve; the unfused --adapter-path route silently served the base model on the version tested.
• Guard every response in code: import the exact training prompt, check for missing content, and validate the record.
• A perfect held-out score only covers data like the training set. Test on messy real inputs, and fix label leakage in the data rather than expecting training to overcome it.