Articles for people who
ship the stack
Rewrites on React, Node.js, TypeScript and AI. A translated body is shown when it exists. If it does not, you see the English text and a note that says so.
Tagged: Inference
Inside an LLM: Pure Python Inference for Qwen3-0.6B on MLX
A step-by-step build of LLM inference in about 200 lines of Python with MLX: tokenizer, RMSNorm, RoPE, grouped query attention, SwiGLU, KV cache and sampling.
13 min readRunning Kimi K3 and Other Big Models Locally with Five Open Repos
Five open source repos that squeeze frontier-scale models onto a laptop or single GPU, using disk streaming, expert caching and offloading, with their real speed numbers.
6 min readvLLM Serve Flags Explained, With Sensible Starting Values
A practical tour of the vllm serve flags you'll actually tune: dtype, memory use, context length, parallelism, API keys and quantization, with defaults to try.
4 min readInference Beyond Text: Tuning VLMs, Embeddings, Speech, Image and Video
Each modality has its own bottleneck. See how to tune VLMs, embedders, ASR, TTS, image and video generators with the right batching, kernels, caching and quantization.
11 min readKV cache is the unseen database eating GPU memory in LLM decode
The KV cache quietly eats GPU memory during LLM inference: prefill vs decode, KV math, GQA and MQA, PagedAttention, quantization and eviction policies.
9 min readHands-On LLM Serving: Ollama, vLLM, and TGI
The serving layer that turns model files into an LLM API: Ollama for local work, vLLM and Hugging Face TGI for GPUs, and what plain transformers lacks.
5 min readllama.cpp vs vLLM vs Ollama: Pick the Local Engine by Bottleneck
llama.cpp, Ollama or vLLM for a local LLM? Match the engine to the bottleneck: laptop portability, friendly DX, or multi-user GPU throughput under load.
5 min readOllama on IBM Z and LinuxONE: Local LLMs Inside the Trust Boundary
Porting Ollama to IBM Z and LinuxONE came down to endian-safe weight loading, so a local LLM can serve regulated data without it leaving the trust boundary.
6 min readLLM Inference End to End: Prefill, Decode, KV Cache, and Sampling
A whiteboard-ready tour of tokenizer through streaming—why prefill is compute-bound, decode is bandwidth-bound, and how GQA, paging, and speculation change the economics.
40 min read
About these articles
Request a 24h estimate
Need the same stack in a production operator layer? Send the brief — estimate within 24 hours.