Local AI for Confidential Documents: What to Run on Your Own Hardware
Keeping contracts, payroll and audit files off cloud APIs: which pipeline stages must be local, what hardware fits, where local models are weaker, and costs.
This article is published in English.
Contracts, payroll, medical records, supplier audits, bank statements: a lot of the documents companies most want to automate are the ones they least want to send to a cloud API. Local AI means the models, the text extraction and the stored results run on hardware the company controls, so the documents never leave it. It is a practical option now, but it comes with trade-offs that are better understood before buying a server.
What "local" has to cover
Running the language model on your own machine is only one part. A document pipeline has several stages, and each of them can leak data if it calls an outside service:
- Text extraction and OCR. Scanned PDFs and photos need OCR. Many convenient OCR options are cloud services; local engines exist and work well on clean scans.
- Embeddings and search. If documents are indexed for search or question answering, the embedding model and the vector index need to be local too.
- The language model. Classification, extraction, summaries and rule checks.
- Logs, queues and backups. Debug logs often contain the full text of a document. They need the same retention and access rules as the documents themselves.
- Telemetry. Some tools report usage to their vendor by default. It should be switched off, and outbound connections from the inference servers blocked.
What hardware you actually need
The short answer: memory decides which models load, and the rest of the hardware decides how fast they run. We measured this on several machines with the same test harness, and the results are in four separate articles:
- RTX 5060 Ti 16 GB: a single consumer card already runs 27B-class models; the fastest setup we measured reached 115.9 tokens per second.
- RTX 5090: the same Qwen3.8-27B model ran anywhere from 67.2 to 253.4 tokens per second depending on the inference engine, which shows how much the software stack matters.
- Mac Studio M5 Ultra vs DGX Spark: two desk-sized machines with large unified memory for models that do not fit on a gaming GPU.
- How much memory for local AI: what each class from 16 GB to 512 GB can run.
For document work, raw speed is rarely the bottleneck. A batch of a few hundred invoices a day is a small load for one decent GPU. What matters more is that the model is good enough on your documents, which you only learn by testing it on them.
Where local models are weaker
The largest cloud models are still ahead on hard reasoning, long messy documents and unusual languages. Local models close much of that gap on narrow, well-defined tasks, especially when the pipeline is designed around their limits:
- let code do what code does well: dates, sums, VAT checks, matching against a supplier list;
- give the model one question at a time instead of "review this contract";
- use structured output (fixed fields) so answers can be validated automatically;
- send uncertain cases to a person instead of forcing an answer.
Whether a given local model is good enough is an empirical question. Build a test set from real documents and compare candidates on it; our notes on evaluations and human review describe how.
A hybrid is often the honest answer
Not every document is equally sensitive. A common design keeps confidential documents on local models and allows a cloud model for public or anonymised material, with the routing rule written down and enforced in code. Another option is a private cloud deployment under a contract that fits your data protection obligations. Which one is right depends on your data, your contracts and your risk appetite, and it should be decided with whoever is responsible for data protection in your company.
Costs to plan for
Local AI moves cost from per-request fees to hardware and operations. Besides the machine itself, plan for power and cooling, someone who updates models and drivers, monitoring, backups and a spare plan for when the box fails. For a single team with a steady document flow this is often cheaper and simpler than it sounds. For spiky or very large workloads the cloud may still win on price.
We build local pipelines for document review and data integration inside client infrastructure. See AI agents for document review and customer data integration, or the regulatory side in AI Act and GDPR for AI automation.