Is a Bigger LoRA Worth It? Measure Accuracy Against Cost, Not Vibes

A folder of thirteen experiments

Picture a folder holding thirteen fine-tuning experiments. They cover style transfer, JSON extraction, tool routing, preference tuning for commit messages, chain-of-thought distillation, and a multi-modal run that reads bar charts. All are LoRA on a 1B Llama, trained locally on an M-series Mac through MLX. Experiment number twelve is the one that changed the workflow. It doesn't teach a model anything new. It only measures the others properly.

The question that gets answered by gut feeling

Take a classification task. It could be routing support tickets. Here it was labeling GitHub pull requests with conventional-commit categories: bug, feat, docs, refactor, test, chore, perf and style. That makes eight classes with clean train and test splits.

You fine-tune a small model and it works. Then comes the question that always comes: would rank 16 beat rank 4?

The usual response is to reason it through. Rank sets the dimensionality of the low-rank update, more rank means more capacity, the task has eight classes and some linguistic subtlety, so probably yes. Then the rank gets bumped, accuracy shifts by an amount that could easily be noise, and the change stays because the time is already spent. That isn't engineering. It's a hunch with a config file attached.

The real gap was that nobody asked whether a bigger adapter paid for itself, and that needs something to divide by. Compare a rank-16 adapter over 16 layers carries about eight times the trainable parameters of rank 4 over 8 layers. It trains longer, takes longer to fuse and is a different artifact to ship. If it buys one accuracy point, that's a poor deal. If it buys fifteen, it's an easy call. Without ever measuring the cost side, there was no way to know which situation applied.

Recipes as plain data

The fix was a harness built on a dull idea, which is the point: a training configuration is data and not code.

RECIPES = [
    {"name": "base-zs",      "train": None, "shots": 0},
    {"name": "base-fs",      "train": None, "shots": 4},
    {"name": "lora-r4-l8",   "train": {"lora_rank": 4,  "num_layers": 8,  "iters": 600}},
    {"name": "lora-r16-l16", "train": {"lora_rank": 16, "num_layers": 16, "iters": 600}},
    {"name": "lora-long",    "train": {"lora_rank": 8,  "num_layers": 8,  "iters": 1500}},
]

Five list entries, nothing more. For each one, the sweep runs everything: it trains the adapter, merges it into the base weights, adds the merged model to Ollama via a generated Modelfile, evaluates it on the held-out test set and records the outcome.

Two of the recipes involve no training. That was deliberate, and it's the idea most worth copying. base-zs is the raw 1B model used zero-shot. base-fs is the same model with four examples placed in the system prompt:

FS_EXAMPLES = (
    "Title: fix crash when checkout receives empty payload\nBody: Guard added.\nLabel: bug\n\n"
    "Title: add bulk delete to dashboard\nBody: New endpoint + UI.\nLabel: feat\n\n"
    "Title: bump dependencies for billing\nBody: Routine maintenance.\nLabel: chore\n\n"
    "Title: speed up search list query with index\nBody: p95 480ms -> 120ms.\nLabel: perf\n"
)

Few-shot prompting is nearly free, costing a longer prompt and around ten minutes to write examples. Fine-tuning brings a training run, a fused model artifact, a deployment story and a commitment to retrain whenever the label set changes. Unless you compare against the free option, you can't claim the expensive one was needed. None of the eleven earlier projects had a few-shot baseline, which looks embarrassing in hindsight.

Four numbers for each recipe

Every recipe gets four measurements, and the value comes from seeing them next to each other:

• Accuracy, which is the reason for doing any of this
• Training time, as wall-clock measured with perf_counter around the real subprocess
• Inference latency, using the median across the full test set and not the mean
• Peak memory, which tells you whether the thing fits on the machine you own

Timing is measured instead of estimated:

t0 = time.perf_counter()
subprocess.run(cmd, check=True)
train_secs = time.perf_counter() - t0

The median is preferred over the mean because a single cold-start outlier at the beginning of a run can drag a mean to a meaningless value, and you'd waste an afternoon chasing a model that's fine.

On an M1 Pro the full sweep takes about 45 minutes, and that's the real payoff. Forty-five unattended minutes is cheap enough that running the sweep is easier than debating it, which changes your default behavior. Opinions about hyperparameters give way to a table.

What the sweep can separate

An unexpected finding had nothing to do with any single number. The lora-long recipe (rank 8, 8 layers, 1500 iterations) exists to split apart two variables people constantly mix up. "A bigger LoRA helped" usually means rank went up and training also ran longer, because the config was changed once and one result was read. Those are separate levers. More rank adds capacity, while more iterations means converging further on the capacity already there. A task can be starved on one and saturated on the other, and the remedy differs completely depending on which.

Hold rank at 8 and raise iterations from 600 to 1500, and the effect is isolated. You pay two and a half times the training time with the same artifact size and the same inference cost. If the result stays near the 600-iteration run, the task was never capacity-limited and rank was a red herring. If it jumps, the model was undertrained all along and the rank debate targeted the wrong knob. No amount of staring at a config file can answer that.

A short closing paragraph in the report is picked by simple rules, naming the recipe with the top accuracy and the one with the best accuracy per second of training. Getting two different winners is the normal outcome. Naming both forces a real decision where "highest number wins" would have quietly made it for you.

A limit worth stating

The harness handles classification, which is the easy case: accuracy is an exact match with a known label, so scoring is just a string comparison at no cost. Generation tasks don't work that way. A comparison of the same kind on a self-improvement loop project had to use an LLM as judge, with exact-match checks only for the structured fields, and then the judge's reliability became its own question.

The structure survives anyway. Recipes stay data, cost is measured on the same clock as quality, and the table keeps its four columns. Only the accuracy function gets harder.

What it comes down to

Most fine-tuning discussion stays on quality, yet most of the choices you actually face are about price. Rank, layers and iterations each cost something different. To compare them fairly, you need a harness so quick to run that nobody bothers to argue instead. A 45-minute unattended job is cheaper than a confident engineer and wrong less often, so make the table first and form the opinion afterwards.