Articles for people who
ship the stack
Rewrites on React, Node.js, TypeScript and AI. A translated body is shown when it exists. If it does not, you see the English text and a note that says so.
Tagged: vllm
vLLM Serve Flags Explained, With Sensible Starting Values
A practical tour of the vllm serve flags you'll actually tune: dtype, memory use, context length, parallelism, API keys and quantization, with defaults to try.
4 min readWhy Ollama Stops Being Enough and What to Run Instead
Ollama is a wrapper around llama.cpp. Learn where it hits limits on load, model formats and AMD GPUs, and when llama.cpp, LM Studio or vLLM fits better.
7 min readFlashAttention and PagedAttention: Two Fixes for Two Different Bottlenecks
Learn how FlashAttention speeds up attention math through tiling and online softmax, how PagedAttention manages KV cache memory, and why vLLM-style stacks use both.
7 min readWhen a Model Never Finishes: Silent Failures Hidden From Hallucination Scores
Two checkpoints, two quantizations and two runtimes tested on invented names: one model stays silent for minutes, and a careful grader exposes what scores miss.
11 min readFixing Ollama Connection Refused on WSL2 and Wiring Up Dify and vLLM
Ollama works in WSL2 but Dify gets connection refused? Trace the bind address, set OLLAMA_HOST, add a portproxy, and avoid the Docker localhost trap when linking vLLM.
7 min readRunning Local Coding Models on a 12GB RTX 3060 With a Model Scheduler
How to pick models that fit a 12GB GPU and why switching between GGUF and SafeTensors backends behind one OpenAI-compatible endpoint turns into a scheduling problem.
5 min readA Model Router on GCP: One Governed API in Front of Multiple Open LLMs
A design for decoupling enterprise apps from individual LLMs: logical tasks, constraint-based model selection, fallback chains, vLLM and a stable OpenAI-style API on GCP.
7 min readWhy Round-Robin Fails LLM Fleets: How llm-d Routes by KV Cache and Queue Depth
Round-robin load balancing wastes KV cache hits and ignores request size. How llm-d scores vLLM replicas on Kubernetes and splits prefill from decode.
10 min readQwen3.8-27B on One RTX 5090: vLLM, LACT Fan Control and Concurrency Benchmarks
Serve Qwen3.8-27B from a headless RTX 5090 with vLLM: Docker GPU setup, Prometheus and Grafana, fixing thermals with LACT, and where throughput saturates under load.
11 min read
About these articles
Request a 24h estimate
Need the same stack in a production operator layer? Send the brief — estimate within 24 hours.