Fine-Tune or Call the API? Costing a Document-Extraction Pipeline

Engineering leaders keep asking the same question, usually right after the first large invoice from OpenAI or Anthropic arrives: should the team fine-tune its own model or keep calling the API? The expected answer is that fine-tuning is cheaper. Sometimes that is true, but typically for narrower reasons than people assume, and for many teams it is simply wrong. This article works through a realistic document-heavy system costed both ways, using list prices from September 2026, so you can see where the savings really come from, when owning a model is justified, and how to sequence the decision without betting eight engineer-weeks on a guess.

The option people imagine no longer exists

One change has quietly reframed the whole debate: at the time of writing, you cannot fine-tune a current frontier model.

According to OpenAI's pricing page, its fine-tuning platform is being phased out: new customers cannot sign up, and o4-mini is the one remaining tunable model, billed at $100 per hour of training. Anthropic's own API has never exposed fine-tuning at all. Its only historical route was supervised fine-tuning of Claude 3 Haiku on Amazon Bedrock, and Haiku 3 has since been retired everywhere except Bedrock and Google Cloud.

So in late 2026, "fine-tuning versus frontier" really means something more specific: take an open-weight model such as Qwen, Llama, a Nemotron variant or gpt-oss, train a LoRA adapter on your data, then host it on your own infrastructure or with a provider like Fireworks or Together. The risks of that choice are not the ones people picture, involving model selection, evaluation and serving that a hosted API normally handles for you. Be precise about it before it reaches a board presentation.

The reference system: an audit-evidence platform

The system costed here is not a toy. It is a complete audit-evidence platform for a mid-sized firm, the sort of product a UK or Nordic accounting practice would actually purchase. It works in seven stages:

1. Intake. Clients upload files through a portal, mostly as scanned PDFs. Typical inputs are invoices, POs, goods-received notes, bank statements, leases and minutes of board meetings.
2. Classification. Each document is assigned a type and sent to the appropriate test.
3. Extraction. Structured fields are pulled into a fixed schema, including vendor, date, net amount, VAT, PO reference, approver, currency and cost centre. The output must be valid JSON that conforms exactly to the schema on every call.
4. Control testing. The three-way match checks whether the invoice agrees with the purchase order and the goods-received note, and whether the approver was within their delegated authority. Most of this is ordinary code rather than a model.
5. Exception triage. Anything that fails a test is scored and ranked.
6. Workpaper drafting. The system writes a first draft of each finding, for example that 42 payments above a £50,000 threshold were tested and three had no documented secondary approval.
7. Auditor Q&A. Auditors query the entire engagement file and receive grounded answers with citations back to the source documents.

Stages 2 and 3 consume almost all of the tokens, and they are also the most repetitive, mechanical and schema-constrained part of the system. Stages 6 and 7 are where real judgement happens, yet their share of volume is negligible. Keep that split in mind, because the rest of the analysis follows from it.

Estimating token volume

The model assumes every document is converted to text with OCR before any model sees it. That avoids image-token pricing and is what you would do in practice anyway.

Each document passes through three calls (classify, extract and self-check), totalling roughly:

• 5,200 input tokens, covering the document text, the schema and some few-shot examples
• 600 output tokens for the structured result

Two volumes are modelled:

• Pilot: 100,000 documents per month, representing one firm during one busy season.
• Scale: 2,000,000 documents per month, representing the same product sold to fifty firms.

That comes to about 580 million tokens a month at pilot volume and 11.6 billion at scale.

What the bill looks like

The figures below use standard-tier list prices with global routing and no batch or caching discounts, as published on each vendor's pricing page in September 2026. Prices change often, so treat them as a snapshot and re-check before budgeting.

At pilot volume, the case for fine-tuning collapses. Running the whole extraction pipeline on Claude Sonnet 5 costs about $1,640 a month, Haiku 4.5 about $820, and a fine-tuned 8B model about $116. Saving roughly $1,500 a month will never repay the effort of building a fine-tune. At this stage, ship on the API.

At scale volume, the choice becomes a genuine one:

• Claude Opus 5: about $82,000 per month
• GPT-5.6 Sol: about $65,600 per month
• Claude Sonnet 5: about $32,800 per month
• Claude Haiku 4.5: about $16,400 per month
• GPT-5.6 Luna: about $3,520 per month
• Fine-tuned 8B, served serverless: about $2,320 per month

It is tempting to headline a saving of more than 90% from fine-tuning, and against the flagship tiers the arithmetic supports it: the fine-tune costs about 7% of the Sonnet 5 bill and under 3% of the Opus 5 bill. But no team should be running bulk invoice extraction on a flagship model in the first place. Measuring an optimized design against a deliberately wasteful one is marketing, not analysis.

Routing, not fine-tuning, delivers the big saving

Compare the last two lines of that list: $3,520 for GPT-5.6 Luna against $2,320 for the fine-tuned 8B model. The difference is $1,200 a month, or about $14,400 a year.

Building the fine-tune properly involves labelling data, writing an evaluation harness, running training, standing up serving and adding drift monitoring. A fair estimate is eight engineer-weeks. Multiply that by a typical blended European contractor rate and, at most rates, the payback from token savings alone stretches well beyond eighteen months.

The more useful conclusion is that the largest single cost reduction comes from routing. Moving classification and extraction off the flagship onto the cheapest tier that still passes your evaluations takes the Opus-level bill from $82,000 to $3,520 a month, a reduction of about 96%. With a router and a solid evaluation set, that change can be made in an afternoon. Fine-tuning on top of it saves a further 34% or so. The same principle, matching model size to each call, is explored more broadly in right-sizing LLMs with routing, retrieval and evaluation.

Training itself is close to free. Fireworks lists LoRA supervised fine-tuning for models up to 16B parameters at $0.50 per million training tokens. Twenty thousand labelled examples of 2,500 tokens each, trained for three epochs, is 150 million training tokens, or about $75 per run. Eight runs during development add up to roughly $600. Compute was never the expensive part; data preparation and evaluation are. Any fine-tuning quote built mainly on GPU cost comes from someone who has not done one.

Why fine-tune at all

If the token bill does not justify it, two other factors can.

Reason one: strict schema adherence

This matters especially in audit work and gets far less attention than it deserves.

Frontier models are generalists. Ask one to pick from 51 fixed subcategories and it will now and then produce a 52nd, since producing plausible-sounding text is precisely its training objective. For a chat assistant, such slips barely matter. In a workpaper that supports a signed audit opinion, it is a defect.

Recent research points the same way, though much of it is preprint work and should be read with that in mind:

• A 2026 preprint on classifying security documents evaluated models against 51 predefined subcategories, with answers required in a rigid JSON format. There, a locally hosted fine-tuned model beat prompted frontier models by 15 to 20 percentage points, and the researchers observed GPT-5 inventing subcategory names absent from the taxonomy. Their interpretation is the important part: frontier models handle loosely formatted extraction well, but under strict schema constraints, calibration matters more than reasoning ability.
• Another preprint reported a fine-tuned Qwen2.5-0.5B, a half-billion-parameter model that fits on one consumer GPU, averaging 0.83 micro-F1 on general-domain relation extraction. With only minimal zero-shot prompting, GPT-5.4 reached 0.69 on the same measure and Claude Sonnet 4.6 reached 0.66. The authors stress that this does not make small models inherently stronger; it shows that adapting a model to a narrow, fixed task can outweigh raw capability. Audit extraction is exactly that kind of task.
• Anthropic's own published results from fine-tuning Claude 3 Haiku showed classification accuracy on a comment-moderation task rising from 81.5% to 99.6%, with 85% fewer tokens per query. The domain differs, but the pattern is the same.

For an audit product, a two-point improvement in field-extraction accuracy may be worth more than all inference spending combined, because each extraction error becomes a manual review, and human review is the costliest resource in the business.

Reason two: knowing exactly where the data lives

In Europe, this is often what decides the deal.

The audit files of a UK or EU accounting firm contain client financial records, employee data and sometimes personal data about third parties. Where processing happens is not a secondary concern; it is typically the second question procurement asks.

The current options, as of September 2026, surprise many teams:

• Anthropic's first-party API has no EU residency option. The inference_geo parameter accepts global or us, with US-only processing priced at 1.1 times the standard rate. To keep Claude inside the EU you have to use an EU region of AWS Bedrock or Google Vertex, where regional endpoints carry a 10% premium over global ones. Claude on Microsoft Foundry had no EU data zone at the time of writing.
• OpenAI supports regional processing, adding 10% to the price for models launched from 5 March 2026 onward.
• Fireworks applies a 1.5x multiplier when a dedicated deployment is restricted to a region.
• Self-operated GPUs in a data centre in, say, Frankfurt or Dublin carry whatever hardware and hosting cost you negotiate, and the data never leaves your control.

With residency priced in at scale volume, the picture shifts. Two dedicated H100s running the fine-tuned 8B model, region-restricted and running around the clock, come to roughly $17,520 monthly. Serving the same workload with Claude Sonnet 5 on an EU Bedrock endpoint would run about $36,080, and with Opus 5 about $90,200.

That is the genuine European argument for owning the model. It is not that tokens are cheaper; it is that the compliance explanation fits in one sentence rather than an architecture diagram.

Do not buy dedicated GPUs too early

A flat monthly GPU cost looks attractive and catches many teams out, because it pays off only at volumes few products ever hit. Taking a pair of dedicated H100s on global pricing, roughly $11,680 per month, as the baseline, the crossover points sit near:

• 700,000 documents per month for Claude Sonnet 5, below which the API is cheaper
• 1.4 million per month for Claude Haiku 4.5
• 6.6 million per month for GPT-5.6 Luna

A typical audit product is likely to sit under every one of those lines for its first two years. Serverless hosting of a fine-tuned model avoids the trap entirely: you get custom weights without paying for idle GPUs, which makes it the sensible starting point. The trade-off is less control over latency and capacity, which matters only once volume is high and steady.

Four questions that settle the decision

1. What is your real monthly token volume? If you process less than about a billion tokens monthly, pick the least expensive frontier tier that clears your evaluations and invest the eight engineer-weeks somewhere more useful. A fine-tune at pilot volume is theatre, not engineering.

2. How narrow and constrained is the task? A fixed output format, a fixed label set and thousands of near-identical examples a day point to fine-tuning. Open-ended reasoning, judgement calls and narrative drafting for a partner's signature belong on a frontier model, and are likely to stay there.

3. Where must the data reside? A contract clause requiring processing to stay within the EEA reshapes the shortlist before cost enters the discussion. Include the residency premium in the first estimate rather than discovering it during architecture review.

4. Can you get at least 10,000 labelled examples? An NVIDIA Research position paper on small language models in agentic systems suggests somewhere between ten thousand and a hundred thousand examples as a working range for tuning a small model. Lacking that, the first project should be instrumenting the frontier pipeline so it records its own training data.

That last point is the most valuable pattern here, and the least promoted. Ship on the frontier API and log every call, including inputs, outputs and reviewer corrections. Six months later you own a labelled dataset that cost nothing extra, and you can fine-tune based on evidence rather than hope. Logging client documents has its own data-protection obligations, so design retention and access rules for those logs from the start.

The same NVIDIA paper also estimated the share of LLM calls that small models could take over in three open-source agent projects: roughly 60% for MetaGPT, 40% for Open Operator and 70% for Cradle. Not all of them and not none of them. The right answer is a mix, and only logs reveal which calls belong where.

The strongest counter-argument

Enterprise adoption data points the other way. According to Menlo Ventures' enterprise survey, the open-source share of enterprise workloads dropped from 19% to 11%, while spending consolidated on closed frontier APIs: Anthropic at 40% of enterprise LLM spend, OpenAI at 27% and Google at 21%. (Menlo is an investor in Anthropic, which is worth knowing when reading those numbers.)

This does not contradict the analysis above; it measures where the friction sits. Closed APIs win on convenience, and convenience usually prevails. Teams that get real value from open-weight fine-tunes chose the operational burden deliberately, for a reason they could state clearly. If you cannot state that reason, the survey is a signal worth heeding.

A brief note on EU regulation

UK and EU teams will ask about the AI Act, so a short summary is useful. Treat it as orientation, not legal advice, and confirm the current text before relying on any date.

The Digital Omnibus on AI, formally Regulation (EU) 2026/1744, appeared in the Official Journal on 24 July 2026 and took effect three days later, on 27 July. It moved the high-risk obligations for standalone Annex III systems back from 2 August 2026 to 2 December 2027. For AI built into products covered by Annex I, the new date is 2 August 2028.

The Article 50 transparency obligations were not postponed and have applied since 2 August 2026, as originally planned. For systems placed on the market earlier, the Article 50(2) marking duty kicks in on 2 December 2026.

Tools that support auditors and keep a human approval step usually sit outside Annex III, but assess your own use case: if any component influences hiring decisions or creditworthiness, it is in scope. None of this affects GDPR, which governs the client data flowing through the system regardless of how the AI Act classifies the system.

Demand is real. In a Wolters Kluwer survey, 39% of 4,214 internal audit professionals said they already use AI, and another 41% expected to start within a year.

A phased plan for a build like this

For a team starting this kind of system, the recommended sequence is:

1. Ship on the cheapest frontier tier that passes your evaluation set. Instrument every model call, let practising auditors use it through an actual busy season, and hold off on any fine-tuning.
2. Study the logs. Identify the two or three call types that account for most tokens and check whether they are repetitive and schema-bound. They nearly always are.
3. Fine-tune only those calls. Host them on serverless infrastructure initially, leave triage, drafting and question answering on the frontier model, and add a router in front of both.
4. Adopt dedicated GPUs only once volume or a contractual residency requirement forces the move, and not a day earlier.

Three of those four phases cost less than what many teams do on day one.

Key takeaways

• Current frontier models generally cannot be fine-tuned, so the real choice is an open-weight model with a LoRA adapter versus a hosted API.
• In document pipelines, a few repetitive, schema-bound calls consume most tokens; cost analysis should start there.
• Routing those calls to the cheapest competent tier saves far more than fine-tuning, and takes hours rather than weeks.
• Fine-tuning earns its place through strict schema adherence and data residency, not through the token bill.
• Training compute is cheap; labelled data, evaluation and operations are the real cost.
• Log production calls from day one so a future fine-tune starts from evidence.
• Prices, residency options and regulatory dates change quickly; re-verify every figure against current vendor pages and official texts before committing a budget.

The underlying question was never small model versus frontier model. It is which of your calls genuinely need to reason. Answer that, and the cost question largely answers itself.