Ollama on IBM Z and LinuxONE: Local LLMs Inside the Trust Boundary
Porting Ollama to IBM Z and LinuxONE came down to endian-safe weight loading, so a local LLM can serve regulated data without it leaving the trust boundary.
This article is published in English.
Sources: **Ollama on IBM Z & LinuxONE (s390x): Enabling Local LLM Inference on the Mainframe**
Why mainframes and local LLMs belong in the same sentence
Mainframes already guard some of the most sensitive enterprise data on earth. The open question is whether they can also host modern language models without shipping that data to external GPU clouds or specialty appliances.
For regulated workloads the answer is not academic. Moving financial, healthcare, or government text off-platform creates compliance exposure, audit drag, and latency. Z and LinuxONE platforms already ship isolation features, cryptography, and operational controls that few commodity clouds match. Local inference extends that trust boundary instead of punching a hole through it.
In summer 2026, work at IBM’s Poughkeepsie lab brought Ollama — a popular local LLM runtime — up natively on s390x. Packaging was not the hard part. Correctness at the bottom of the stack was: reading little-endian model weights on a big-endian machine. Once that path was solid, the rest of the port followed.
What had to change
Ollama on other architectures assumes little-endian layouts for GGUF tensors and related metadata. On big-endian s390x, naive loads scramble floats and integers. The fix is disciplined byte-swapping at tensor load boundaries (and anywhere else on-disk structures encode multi-byte values), plus validation that embeddings and logits match a known-good little-endian run within tolerance.
Build and CI had to cross-compile or native-build for s390x, exercise model pull/run, and catch endian bugs before they looked like “the model is dumb.” GPU offload stories differ on Z versus discrete NVIDIA cards; CPU inference with enterprise crypto and isolation may still win for sensitive corpora even when tokens/sec trail a gaming GPU.
Running it
After the port, the familiar Ollama commands apply on LinuxONE/Z Linux:
oc label pvc <pvc-name> opendatahub.io/dashboard=true -n <namespace>
Operators pull a model, run a prompt, and keep weights inside the same regulated zone as the source documents. That is the product win: inference stays where the data already is.
Who should care
- Banks and insurers with Z-centric systems of record
- Public sector shops barred from public LLM APIs
- Teams that already standardized on Ollama elsewhere and want one operator UX on-prem
Local LLMs on s390x are not a curiosity demo. They are a way to keep the inference trust boundary aligned with the data trust boundary you already paid to build.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.
Operational note: treat endian-aware loaders as security-relevant code. A missed swap can look like random quality loss and waste days of “prompt tuning” when the real bug is bytes on the wire from disk.