Hands-On LLM Serving: Ollama, vLLM, and TGI

The serving layer that turns model files into an LLM API: Ollama for local work, vLLM and Hugging Face TGI for GPUs, and what plain transformers lacks.

Publisher: REACTAPP.TOPAdded to this site Sep 26, 20265 min read1118 words

This article is published in English.

Sources: Serving LLMs, Hands-On — Ollama, vLLM & TGI

Serving engines: the layer under “just call the model”

You have been calling models — often through Ollama. This walkthrough is about the inference server: it loads weights, manages GPU memory, batches work, and streams tokens so many clients can hit one API. Three names you will keep hearing: Ollama, vLLM, and TGI (Text Generation Inference).

Ollama experiments run fine on a laptop CPU. vLLM and TGI want a GPU — Colab or Kaggle work if you lack one. Treat this as a work-along: run the blocks, do the exercises.

Why plain `transformers` is not a server

Loading a model and calling .generate() works for one slow request. Under load it falls apart: no continuous batching, weak memory planning, awkward multi-user streaming. Serving engines exist to make “model file” into “API.”

Ollama — local first

Install, pull, run, and hit the HTTP API:

ollama pull llama3.2
ollama serve            # usually already running; serves on http://localhost:11434

from openai import OpenAI   # pip install openai

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key is ignored locally

resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Explain KV cache in one sentence."}],
)
print(resp.choices[0].message.content)

Use it to learn prompts and build apps without cloud keys. Know its limits when you need fleet-scale scheduling.

vLLM — batch-friendly GPU serving

On a GPU runtime:

pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct       # OpenAI-compatible server on :8000

OpenAI-compatible endpoints and continuous batching are the point. Watch GPU memory and max context flags.

TGI — Hugging Face’s serving stack

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct

TGI focuses on production HF model serving with streaming and tracing hooks. Compare latency and ops complexity against vLLM for your models.

Side-by-side mental model

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")   # ← base_url changed
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",                           # ← model changed
    messages=[{"role": "user", "content": "Explain KV cache in one sentence."}],
)
print(resp.choices[0].message.content)

Ollama: DX and local iteration
vLLM: tokens/sec on NVIDIA under concurrency
TGI: HF-centric production serving

Your turn

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v ~/.cache/huggingface:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct

Hit each server with the same prompt, note TTFT and tokens/sec, then write which engine fits your next project — laptop demo, internal GPU box, or multi-tenant API.

Keep the deployment boring: compose up, health checks, and a single place to rotate models. Boring is how internal tools survive contact with real on-call schedules.

← All articles Send the brief