Cutting Ollama Cold Starts on Runpod Serverless from 50s to 4.4s

Serverless GPUs have one ugly habit. When a worker spins up from zero and runs Ollama, the first token can take anywhere from 30 to 60 seconds. For a chat you poke at once, that's merely irritating. In a production pipeline it breaks the whole idea. Here's how a 27B hybrid Mamba/attention model on Runpod Serverless went from a 50-second cold start to 4.4 seconds, and, more useful, how each bottleneck got found, since the investigation is the part most write-ups skip.

The setup

• Model: a 27B-parameter hybrid, mostly linear-attention (Mamba-style) layers with full attention on every fourth layer, plus a bundled vision encoder that this workload never touches. It's run as qwen3.8:27b.
• Platform: Runpod Serverless on H100-class GPUs, with the GGUF weights (Q4_K_M quantization, roughly 16 GB) sitting on a persistent network volume. The H100 is more than the model needs, since it fits in about 18 GB of VRAM at Q4. The spare headroom is deliberate, so a larger model can move in once cold starts are cheap.
• Baseline: the stock runpod-workers/worker-ollama image, with nothing changed.

Step one: the stock image

Straight out of the box, the log said:

llama-server started in 50.50 seconds

That's tolerable on a laptop and costly on serverless, where every second of cold start is billed. The container logs show where the time went:

• Loading weights by mmap from the network volume: roughly 9 to 14 seconds.
• Warming the main model with an empty forward pass: about 23 seconds.
• Loading the vision encoder and warming it separately: around 2 seconds.
• Speculative decoding and other initialization: about 3 seconds.

The 23-second warmup stands out immediately, and llama.cpp even prints how to skip it:

warming up the model with an empty run - please wait ... (--no-warmup to disable)

The catch is that Ollama's Go wrapper never passes --no-warmup to the llama-server process it launches. The option exists in the engine. Ollama just doesn't expose it.

Step two: patch Ollama instead of replacing it

The tempting move is to bypass Ollama's orchestration and launch llama-server yourself. That turned out to be a trap. Ollama's Go code does a lot of real work before spawning the server: detecting the GPU backend, building LD_LIBRARY_PATH, and choosing the right GGML_BACKEND_PATH for the correct CUDA variant. Redoing all that by hand kept landing in CPU fallback, which showed up as the warning that no usable GPU was found and that the gpu-layers option would be ignored, plus roughly a fivefold drop in tokens per second.

What worked was a single-line patch to Ollama's own source, which leaves its tested logic to do the rest:

// llm/llama_server.go
params := []string{
    "--model", launch.modelPath,
    "--port", strconv.Itoa(port),
    // ...
}
// Custom patch: skip llama-server's ~23s empty-run warmup when
// OLLAMA_SKIP_WARMUP=1 is set.
if os.Getenv("OLLAMA_SKIP_WARMUP") == "1" {
    params = append(params, "--no-warmup")
}

The build is cheap. llama-server isn't recompiled at all. It stays a prebuilt binary lifted from the official ollama/ollama Docker image, so there's no CUDA or C++ compilation. Only the small Go wrapper is rebuilt, which is quick and low risk.

The result was 37.2 seconds. That's a genuine win of about 5 seconds from this patch, nowhere near what removing 23 seconds should have delivered. Something else was eating time.

Step three: a mystery gap of 32 seconds

After the warmup patch, the logs still showed a silent stretch:

srv load_model: loaded multimodal model, '...'
                     [ ~32 seconds of nothing ]
cmn common_conte: the context does not support partial sequence removal

It lasted 32 to 33 seconds on every run. That consistency was the real clue. Honest variable computation doesn't repeat to the decisecond across different configurations, so a number that never moves smells like a fixed cost.

Three separate theories each got tested with evidence, and all three were dead ends:

1. Speculative decoding through multi-token prediction. The model has built-in MTP layers that Ollama enables automatically. Disabling them with OLLAMA_DISABLE_MTP was confirmed by the log reporting no speculative-decoding implementations, yet the gap barely moved, from 37.2 to 36.4 seconds.
2. The prompt cache. A newer llama.cpp feature pre-allocates an 8 GB prompt-cache arena. Turning it off with --cache-ram 0 was confirmed by a "prompt cache is disabled" message, and the time stayed at 36.4 seconds.
3. Vision encoder warmup. A line reading get_dummy_batch: warmup with image size = 1472 x 1472 appeared just as the gap began. That's a separate warmup for the vision model, which --no-warmup doesn't cover because it only gates the text model. Since the vision encoder isn't needed here, Ollama was patched to skip loading the multimodal projector entirely. No vision log lines remained, but the total was still 36.2 seconds.

Four changes in, the same figure of roughly 36 seconds kept coming back, unmoved even after removing whole subsystems. That's the sign you're looking at the wrong layer.

Step four: CUDA JIT compilation

The model's hybrid design uses unusual fused kernels. Right before the gap, the logs print a line resolving fused ops for DeepSeek V4 HC and Gated Delta Net support. Those aren't ordinary transformer kernels, and they're exotic enough that the CUDA driver probably hasn't compiled them for this exact GPU before.

CUDA supports that case. Kernels ship as PTX, an intermediate representation, and get compiled just in time into native machine code (SASS) the first time they run on a given GPU architecture. Compiled output does get cached, yet by default that cache lives only as long as the process does. Each serverless cold start is a brand-new process, so nothing ever carries over.

The solution mirrors a trick used earlier for vLLM's torch.compile cache: point the cache at persistent storage.

# start.sh
export CUDA_CACHE_PATH=/runpod-volume/cuda-jit-cache
mkdir -p "${CUDA_CACHE_PATH}"

/runpod-volume is the same network volume that stores the weights, so the cache outlives every future cold start, on any machine Runpod puts the worker on, not only the one that built it.

The first request after deploying, with an empty cache, still takes 36.4 seconds. There's no free lunch on run one. The second request, with a warm cache, reports:

llama-server started in 4.36 seconds

The whole progression

• Official image, unchanged: 50.5 seconds, the baseline.
• Custom build with only structural changes (prebuilt binary, no Ollama patches yet): 41.9 seconds.
• Add the no-warmup patch: 37.2 seconds, because the main-model empty run is skipped.
• Add the CUDA JIT cache, first run: 36.4 seconds, since the cache is still being built.
• Add the CUDA JIT cache, every run afterward: 4.4 seconds, and it stays that way.

Generation stayed fully on the GPU throughout, with no regressions: prefill ran at about 2,300 tokens per second and decoding at about 75 tokens per second. Measured end to end from container boot to ready for the first token, a warm cache gives roughly 12 to 15 seconds, down from around 50 to 65 seconds originally.

Lessons if you try this

• Read the llama-server lines Ollama prints. They often name the exact flag you need, and both --no-warmup and --cache-ram 0 were spelled out in plain text.
• Patch one line rather than rebuilding subprocess orchestration. Ollama's Go code does work, like GPU backend selection, that's easy to get subtly wrong when copied.
• A gap that won't shrink however much you disable means you're solving the wrong layer. Time got wasted on MTP, the prompt cache and vision before it became obvious that the constancy of the number was the hint worth trusting sooner.
• Anything normally cached per process, such as compiled kernels and JIT artifacts, can usually be redirected to disk. CUDA's JIT cache and PyTorch's torch.compile cache both work this way. If your platform has persistent storage, use it for them.

Costs to be honest about

The setup means keeping a small patch against Ollama's Go source and rebuilding it, which adds a few minutes with every Ollama version bump. It isn't zero-maintenance. The build came from Ollama's main branch to get support for this new model architecture, so a stable tagged release might behave a little differently. And the custom image doesn't get the fleet-wide layer caching that the popular official image enjoys, so first-time image pulls on a brand-new machine remain a separate cost.

None of that outweighs going from 50 seconds to 4.4.