Home / Articles / LLMOps for Small Language Models: Serving Frameworks and Production Playbooks

This article is published in English.

LLMOps for Small Language Models: Serving Frameworks and Production Playbooks

Why SLMs win on cost and privacy, how vLLM, SGLang, TGI, llama.cpp, Ollama, WebLLM, ONNX, and TensorRT-LLM compare, and how to quantize, evaluate, and route them in production.

4127 words

Hands-on LLMOps for compact language models: when on-prem or on-device inference beats frontier APIs, which serving stacks match which hardware limits, and how to keep them healthy after the first successful curl.

Introduction

Somewhere between “fine-tune the largest model for everything” and “run a sub-billion-parameter model on a phone,” production priorities shifted. Many tasks—intent classification, tool routing, structured extraction, RAG re-ranking—do not need frontier-scale models once a solid SLM is quantized and served well.

The catch: an SLM without a serving strategy is only a checkpoint on disk. Running one in production raises GPU memory, batching, quantization fidelity, rollback, and evaluation questions that classic MLOps playbooks barely touch.

A model you cannot deploy, monitor, or roll back is not a production asset—it is a liability with attractive benchmark scores.

This guide covers the problem, the framework landscape, and a production playbook—in that order. The pipeline is continuous: a checkpoint becomes a capability only after it passes serving, evaluation, and operational gates.

The Problem: Why “Bigger Model” Stopped Being the Default Answer

1. Cost and latency compound at scale

Sending a simple ticket-classification query to a 70B-class model wastes capacity. Extra parameters buy capability you may not need while multiplying GPU-seconds and tail latency under concurrency. At product scale those costs dominate.

2. Data gravity and privacy push inference to the edge

Healthcare, finance, and on-device consumer workloads often cannot ship raw text to a third-party API. An SLM that fits in a few gigabytes of RAM can live beside the data—on-prem GPUs, laptops, or phones—meeting residency requirements the cloud frontier model cannot.

3. Serving infrastructure has its own failure modes

Model serving fails unlike ordinary app bugs: GPU memory fragmentation from naive KV-cache allocation, scheduler starvation under bursty traffic, quantization mismatches that silently degrade quality, and cold starts that blow SLOs after scale-to-zero.

4. LLMOps is a distinct discipline from MLOps

Classic MLOps assumes relatively stable shapes and deterministic scores. LLMOps adds non-deterministic text, prompt and tool versions, token economics, and safety filters. Regression is no longer only “accuracy dropped 2%”—it is also “JSON schema adherence fell” or “p95 tokens/sec collapsed after a driver update.”

What is LLMOps, and How is it Different From MLOps?

LLMOps covers how teams ship, host, pin, watch, and improve language models in live systems—with the operational seriousness given to any other critical service.

Where MLOps asks whether accuracy regressed, LLMOps also asks:

Is the serving engine using GPU memory efficiently under concurrency?
Does the quantized artifact stay faithful enough for this task?
Are prompt/tool versions pinned and rollbackable?
Can traffic shift across models without client rewrites?
Do traces expose token cost and latency per route?

The rest of this guide answers those questions for SLMs specifically.

The SLM Deployment Framework Landscape

For raw GPU throughput, vLLM is a common high-QPS choice with an OpenAI-compatible API on NVIDIA or AMD GPUs. SGLang competes when prefix reuse and structured generation matter. Hugging Face TGI fits teams already standardized on the Hub and Helm charts.

On the portable side, llama.cpp / GGUF runs on CPUs, Apple Silicon, and many consumer GPUs. Ollama wraps that stack for a two-command local experience. LM Studio adds a desktop GUI for evaluation. MLC-LLM / WebLLM compile for browsers and mobile. ONNX Runtime GenAI targets Windows/NPU and enterprise ONNX standards. TensorRT-LLM + Triton chase maximum NVIDIA fleet performance after ahead-of-time compile.

Each turns a checkpoint into responses; they differ on hardware assumptions, ops weight, and concurrency design.

Framework Deep Dives

vLLM

vLLM popularized PagedAttention, managing the KV cache more like virtual memory so concurrent sequences waste less GPU RAM. Continuous batching keeps utilization high.

# Install vLLM
pip install vllm

# Serve using VLLM
vllm serve Qwen/Qwen2.5–1.5B-Instruct - port 8000 # fully OpenAI-compatible endpoint


# Make Request
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
  "model": "Qwen/Qwen2.5–1.5B-Instruct",
  "messages": [{"role": "user", "content": "Write a haiku about model quantization"}]
}'

# Python Script for vllm

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="Qwen/Qwen2.5–1.5B-Instruct",
    messages=[{"role": "user", "content": "Summarize LLMOps in one sentence."}],
)
print(resp.choices[0].message.content)

Best for: high-QPS backends and RAG services needing OpenAI-compatible endpoints. Pros: throughput, model coverage, quantization formats (AWQ/GPTQ/FP8). Cons: GPU-centric; still needs orchestration for HA.

SGLang

SGLang centers on RadixAttention for sharing prefix caches across related requests—strong for agent loops with repeated system prompts—and includes an sgl.function DSL for structured generation.

# Installation
pip install "sglang[all]"

# SGLang launch server
python -m sglang.launch_server \
 - model-path Qwen/Qwen2.5–1.5B-Instruct \
 - port 30000


# CURL Request
curl http://localhost:30000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
  "model": "Qwen/Qwen2.5–1.5B-Instruct",
  "messages": [{"role": "user", "content": "Write a haiku about model quantization"}]
}'

import sglang as sgl

@sgl.function
def classify(s, ticket):
  s += sgl.user(f"Classify this support ticket as billing/technical/other: {ticket}")
  s += sgl.assistant(sgl.gen("label", max_tokens=8))

sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))
state = classify.run(ticket="My invoice charged me twice this month")
print(state["label"])

Best for: agent pipelines and constrained JSON outputs. Pros: prefix reuse, structured output. Cons: smaller ecosystem than vLLM; GPU-only.

Hugging Face Text Generation Inference (TGI)

TGI plugs into the Hub ecosystem with Docker/Kubernetes-friendly packaging.

docker run - gpus all -p 8080:80 \
-v $PWD/data:/data \
ghcr.io/huggingface/text-generation-inference:latest \
 - model-id microsoft/Phi-3.5-mini-instruct \
 - quantize bitsandbytes-nf4

from huggingface_hub import InferenceClient
client = InferenceClient("http://localhost:8080")
print(client.text_generation("Explain quantization in one line.", max_new_tokens=64))

Best for: Hub-centric teams and regulated stacks wanting a supported serving path. Pros: Hub integration, quantization options, Helm story. Cons: some workloads still prefer vLLM on raw throughput; watch license terms over time.

llama.cpp / GGUF

Born to run LLaMA on a MacBook CPU, llama.cpp underpins much of the local/edge SLM world via GGUF quantization.

# macOS: brew install llama.cpp   |   or build from source:
# git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build

# Pull a quantized SLM straight from Hugging Face and serve an OpenAI-compatible endpoint
llama-server -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M \
    --port 8090 -c 4096 -ngl 999   # -ngl offloads layers to GPU if available (Metal/CUDA)

curl http://localhost:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"What is GGUF?"}]}'

Best for: CPU servers, Apple Silicon, Pi/edge boxes without CUDA. Pros: portability, mature quant ladder, MIT. Cons: concurrent throughput trails GPU-native stacks.

Ollama

Ollama wraps llama.cpp into a pull-and-run developer experience with a local HTTP API.

ollama pull qwen2.5:1.5b
ollama run qwen2.5:1.5b "Write a haiku about model quantization"

import requests
r = requests.post("http://localhost:11434/api/chat", json={
  "model": "qwen2.5:1.5b",
  "messages": [{"role": "user", "content": "Give me 3 LLMOps metrics to track"}],
  "stream": False,
})
print(r.json()["message"]["content"])

Best for: local dev, prototypes, small-team self-hosting. Pros: DX and defaults. Cons: not aimed at high-concurrency production scheduling.

LM Studio

A desktop GUI over similar local engines for browsing, downloading, and chatting—plus a one-click OpenAI-compatible server for demos. Ideal for evaluation before writing deployment manifests; not an infra component you fleet.

MLC-LLM / WebLLM

Built on Apache TVM, MLC compiles models for many backends; WebLLM runs fully in-browser.

pip install mlc-llm

mlc_llm chat HF://mlc-ai/Qwen2.5-1.5B-Instruct-q4f16_1-MLC

// WebLLM: fully in-browser inference, no backend server
import * as webllm from "@mlc-ai/web-llm";

const engine = await webllm.CreateMLCEngine("Qwen2.5-1.5B-Instruct-q4f16_1-MLC");
const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Explain WebGPU inference simply." }],
});
console.log(reply.choices[0].message.content);

Best for: mobile apps, privacy-preserving client inference, offline features. Pros: cross-target compile; no server for WebLLM. Cons: compile complexity; device limits in browsers.

ONNX Runtime GenAI

Microsoft’s GenAI API extends ONNX Runtime for autoregressive generation with KV-cache management across Windows DirectML and NPUs.

pip install onnxruntime-genai

python -c "
import onnxruntime_genai as og
model = og.Model('phi-3.5-mini-onnx-directml')
tokenizer = og.Tokenizer(model)
tokens = tokenizer.encode('Explain ONNX Runtime GenAI briefly.')
params = og.GeneratorParams(model)
params.set_search_options(max_length=200)
generator = og.Generator(model, params)
generator.append_tokens(tokens)
while not generator.is_done():
    generator.generate_next_token()
print(tokenizer.decode(generator.get_sequence(0)))
"

Best for: Windows-native apps and ONNX-standardized enterprises. Pros: portability after export. Cons: conversion friction; smaller community than PyTorch-native engines.

NVIDIA TensorRT-LLM + Triton Inference Server

TensorRT-LLM ahead-of-time compiles engines with fused kernels; Triton hosts multi-model fleets.

# Build a TensorRT-LLM engine for a small model (simplified)
trtllm-build --checkpoint_dir ./qwen2.5-1.5b-checkpoint \
    --output_dir ./qwen2.5-1.5b-engine \
    --gemm_plugin float16

# Serve via Triton
tritonserver --model-repository=/models

Best for: maximum NVIDIA fleet throughput and latency-critical paths. Pros: peak performance once compiled. Cons: engines are SKU- and shape-specific; rebuilds follow hardware or batch-profile changes.

Choosing a Framework: A Decision Guide

A Mental Model: Three Questions, Not a Feature Checklist

Question 1: Where must the model physically run? Browser or phone without a network call → WebLLM/MLC. CPU/Apple/edge without CUDA → llama.cpp/Ollama. Only then consider GPU data-center engines.

Question 2: Is this production serving or human exploration? Exploration → LM Studio or Ollama. Production API → vLLM/SGLang/TGI/TensorRT depending on the next question.

Question 3 (GPU production only): What is the traffic shape, and how much engineering budget exists? Independent one-shot completions → vLLM as default. Highly repetitive agent prefixes / structured outputs → SGLang. Peak NVIDIA optimization with platform investment → TensorRT-LLM + Triton. Hub/Helm comfort → TGI.

Compressed lookups:

Max GPU API throughput → vLLM (or SGLang if agentic/repetitive)
Absolute NVIDIA peak with compile budget → TensorRT-LLM + Triton
CPU/Apple/edge → llama.cpp; DX wrapper → Ollama
Browser/offline client → WebLLM/MLC
Windows/NPU/ONNX standard → ONNX Runtime GenAI
Click-to-eval → LM Studio

Enterprise Agentic Systems vs. Everything Else

Single-turn request/response traffic loves continuous batching on vLLM. Enterprise agents may call the model tens of times per task—planning, tool choice, observation, replan—so prefix caching and structured outputs dominate. Those deployments often land on SGLang or TensorRT-LLM + Triton on self-hosted GPUs, with TGI as an alternative when Hub integration outweighs peak throughput.

Multi-tenant agent platforms also need per-principal quotas, trace baggage across tool calls, and clear rollback of prompt+model pairs together—LLMOps concerns that sit above any one engine.

Production Deployment Guide

Getting something running is roughly a fifth of the work. The rest is keeping it correct, cheap, and safe.

Treat quantization, registry, evaluation, canaries, and routing as one loop every model version travels:

1. Quantization strategy. Default toward AWQ-4bit or GGUF Q4_K_M in many production SLM paths—typically within a couple percent of FP16 task metrics when evaluated carefully—and confirm the chosen engine supports the format. Quantization is not a one-shot conversion; it ends at an eval gate, not at the convert command.

2. Model versioning and registry. Treat fine-tunes and quantized artifacts as immutable, versioned objects—never overwrite in place. MLFlow, Hugging Face Hub repos with digests, or an internal OCI registry of artifacts all work if digests are pinned in deploy manifests.

3. Evaluation gates. Maintain a golden set for the task: classification F1, extraction field accuracy, schema validity rate, refusal correctness, and latency budgets. Block promote-to-prod when gates fail even if demos look prettier.

4. Serving topology. Separate CPU tokenizers/preprocess from GPU workers when helpful; autoscale on queue depth and GPU util, not only RPS. Keep a warm pool if cold starts violate SLOs.

5. Observability. Export request latency, tokens in/out, cache hit rates, batch size, and OOM/retry counts. Sample prompts carefully under privacy policy.

6. Rollbacks. Blue/green or canary on model digest + prompt version as a unit. Instantly revert both when quality spikes.

7. A/B Testing & Multi-Model Routing

Route by task: SLMs for classification and extraction; larger models for open-ended generation. Shadow traffic to candidates before cutover. Track cost per successful task, not only tokens, so a cheaper model that retries twice does not “win” falsely.

Feature flags should bind client routes to named model endpoints behind the gateway so swaps do not require app releases.

Domain Playbooks

Consumer / Mobile Apps

Prefer on-device or near-device SLMs via MLC/WebLLM or GGUF on device runtimes. Budget memory carefully; stream tokens for UX; keep a cloud fallback for hard queries with clear consent.

Other domains (support copilots, internal RAG, edge gateways) follow the same template: pick hardware reality first, then engine, then quantization+eval loop.

Anti-Patterns

Shipping FP16 “because quality” without measuring a quantized baseline on the real task
Using Ollama/LM Studio topologies for high-QPS production without a serving engine built for concurrency
Overwriting model files in place so rollbacks become archaeology
Evaluating only on vibe checks instead of golden sets
Ignoring KV-cache and batch metrics until GPUs OOMs in production
Treating prompt changes as free while pinning model versions—or the reverse

Key Takeaways

SLMs win when tasks are narrow, data cannot leave the premise, or cost/latency budgets forbid frontier models.
LLMOps extends MLOps with serving efficiency, quantization fidelity, prompt/tool versioning, and token economics.
Choose engines by placement (device/CPU/GPU), stage (explore vs serve), and traffic shape (one-shot vs agentic).
Production success is a loop: quantize → register → evaluate → canary → observe → roll back.
Pair durable model digests with doorbell-style routing so clients stay stable while backends evolve.

References

Consult each project’s official documentation for install flags and version pins: vLLM, SGLang, Hugging Face TGI, llama.cpp, Ollama, LM Studio, MLC-LLM/WebLLM, ONNX Runtime GenAI, and NVIDIA TensorRT-LLM/Triton. Hardware guides from NVIDIA and Apple Metal docs complement engine READMEs when tuning batch sizes and quantization.

Keep an internal runbook that records which digests passed which golden sets on which GPU SKUs—the artifact that turns this landscape guide into an operable platform.

Appendix: Operating SLMs Week Two Through Week Twenty

After the first successful curl against a local server, the hard questions arrive: who owns on-call, how are CUDA driver upgrades scheduled, and what happens when a marketing campaign triples QPS overnight. Answer them in writing before the first canary.

Capacity planning for SLMs still needs headroom for KV-cache growth with context length. A 1.5B model that looks tiny at 2k context can pressure memory when clients open 32k windows. Track p95 context tokens separately from request rate.

Security reviews should cover model supply chain: checksum verification on download, who can publish to the registry, and whether system prompts with secrets ever land in client bundles. On-device models need update channels as careful as mobile app releases.

Cost reviews should compare total cost of ownership: GPU rental, engineer time for TensorRT rebuilds, and quality incidents from aggressive quantization. Sometimes a slightly larger SLM at 8-bit is cheaper than a tiny model that forces human escalations.

Training the team matters. Give application engineers a paved road: a Helm chart or Compose file, a standard OpenAI-compatible base URL, and dashboards already wired. Give ML engineers a paved road for publishing digests that pass gates. Most failures happen in the handoff between those groups.

Finally, revisit the “bigger model” temptation quarterly with data. If an SLM’s golden-set score stalls while user tasks grow harder, graduate selectively—per route—not by replacing the entire estate overnight. LLMOps for SLMs is as much about restraint as about acceleration.

Operational note: pin exact wheel versions for CUDA stacks in lockfiles, and record the driver version beside each successful canary so regressions can be bisected across software and firmware layers during incidents.

All articles Send the brief