Self-Hosting Ollama and Open WebUI in Docker for a Team

Local AI setups usually start with one person, one workstation and one happy afternoon. Someone installs Ollama, pulls a model, and the chat output is great. Then a colleague wants the same thing. The fast answer is to copy model files over on a USB stick, install Ollama on a second box, and write down which commands to run. A month on, there are four machines, three Ollama versions, mismatched model files and a shared spreadsheet that maps IP addresses to ports.

That arrangement doesn't hold up. It burns disk, lets configuration drift, and quietly makes whoever built the first machine an unpaid sysadmin for a brittle service. What follows replaces it with a containerized stack where Ollama runs on an internal network, Open WebUI handles logins, and a reverse proxy faces the outside world.

Why hand-sharing fails

The problems tend to show up only after a few weeks of use.

Duplicated storage. A 7B model at Q4 quantization takes around 4 GB. A 70B model needs closer to 40 GB. Copying those files to every team machine fills NVMe drives and floods the local switch.

Version drift. If one person runs an older runtime and another runs a newer build, behavior splits. Models may fail without an error, or two people can get different output for the same prompt. Debugging then turns into a hunt for software versions instead of root causes.

Network exposure. This is the most dangerous one. The common shortcut is setting OLLAMA_HOST=0.0.0.0 and opening port 11434 on the firewall. That puts the raw execution API on the network with no authentication, no encryption and no rate limiting. Anyone who can reach the address can run untrusted workloads, burn compute, read active context, or tamper with stored weights.

A containerized setup removes all three by making the environment declarative. A Docker Compose file becomes the exact specification. Model data lives in dedicated persistent volumes, network access is segmented, and the stack comes up the same way on any Linux host.

The layout

Raw model execution stays separate from anything users touch. Ollama runs only inside an internal bridge network. Open WebUI sits in front of it and handles authentication. Picture it as layers from the bottom up:

• A Linux host with the NVIDIA driver or ROCm
• A Docker bridge network named ai_network (172.28.0.0/16)
• An Ollama container (the inference engine, port 11434 internal, with the GPU passed through directly)
• An Open WebUI container (web app plus database, port 8080 internal, authentication enforced)
• A named volume ollama_models for Ollama and one called open_webui_data for Open WebUI
• Open WebUI published on 127.0.0.1:3000 only, with a reverse proxy (Caddy or Nginx with TLS) in front, and authenticated users behind that

What each piece owns

Each layer needs clear boundaries before anything is wired together.

Ollama is only the backend inference runner. It loads weights into memory, spreads tensor operations over the GPU or CPU, and serves an HTTP REST API. It deliberately has no user management, no sessions and no multi-tenant authorization.

Its state lives entirely in the model store defined by OLLAMA_MODELS, which defaults to /root/.ollama inside the container. Pulling a model downloads layers straight into that volume. Because Ollama keeps nothing else, you can delete or replace the container image and every downloaded model stays on disk.

Open WebUI is the web app and orchestration layer that people actually use. It stores credentials, keeps chat history, enforces role-based permissions and runs Retrieval Augmented Generation pipelines. It does no model math itself. It turns browser actions into API calls to the internal Ollama service. Its own state sits in an embedded SQLite database and local files inside the open_webui_data volume. Losing that volume hurts accounts and conversation history but leaves model binaries alone.

Docker supplies isolation, network policy and a reproducible bootstrap. Because limits and volume mounts live in one manifest, system dependencies stay independent of whatever the host OS has installed.

Why not a desktop app

People often compare headless services with desktop GUI tools like LM Studio. LM Studio is a fine playground for a single researcher trying out prompts. It doesn't offer native multi-user token queuing, background service management or modular microservice integration. Ollama in Docker supplies the daemon behavior, a unified API and the clean separation that a shared deployment calls for.

Picking hardware

Local generation speed depends mainly on VRAM bandwidth, memory bus width and whether the model fits, not on headline FLOPS. These rough pairings are a starting point.

• 7B to 8B models at Q4_K_M need about 6 GB of VRAM. An RTX 3060 or RTX 4060 8GB should deliver around 60 to 85 tokens per second.
• 13B to 14B models at Q4_K_M need about 10 GB. An RTX 4070 or an RTX 3080 12GB should reach roughly 40 to 60 tokens per second.
• 32B to 35B models at Q4_K_M need about 22 GB. An RTX 3090 or RTX 4090 24GB should manage around 25 to 40 tokens per second.
• 70B models at Q4_K_M need about 48 GB, for example two RTX 3090 cards or an RTX 6000 Ada, at roughly 15 to 25 tokens per second.

GPU passthrough prerequisites

On Linux, installing standard GPU drivers on the host doesn't let containers use the GPU. The official NVIDIA Container Toolkit has to be installed so CUDA calls can reach the kernel from inside the container.

On AMD hardware, use the ROCm tag of the image (ollama/ollama:rocm) and pass the /dev/kfd and /dev/dri device nodes into the container.

Docker Desktop on Windows with WSL2 adds its own virtual network bridge quirks. Symptoms include containers that can't be reached or socket errors, and they usually point to host-to-container routing problems in that bridge.

Writing the Compose file

Start by creating a working directory on the server:

mkdir -p ~/local-ai && cd ~/local-ai

Next comes the docker-compose.yml. Current Compose specifications deprecate the top-level version key, so leave it out. The Ollama service comes first:

services:
  ollama:
    image: ollama/ollama:0.6.5
    container_name: ollama
    restart: unless-stopped
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_MAX_LOADED_MODELS=1
      - OLLAMA_NUM_PARALLEL=2
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    networks:
      - ai_network
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Open WebUI follows in the same file:

  open-webui:
    image: ghcr.io/open-webui/open-webui:v0.5.20
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
      - WEBUI_SECRET_KEY=e8f901c23a4b5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0
    ports:
      - "3000:8080"
    volumes:
      - open_webui_data:/app/backend/data
    networks:
      - ai_networkvolumes:
  ollama_models:
    name: ollama_models
  open_webui_data:
    name: open_webui_datanetworks:
  ai_network:
    name: ai_network
    driver: bridge

A few settings in those blocks deserve a closer look.

• OLLAMA_KEEP_ALIVE=24h keeps loaded model tensors resident in VRAM through long idle stretches, so active users don't wait for a reload.
• OLLAMA_MAX_LOADED_MODELS=1 lets only one model occupy VRAM at a time. That avoids fragmentation and out-of-memory crashes.
• 127.0.0.1:11434:11434 ties Ollama to the host's loopback adapter. Outside traffic can't see the API, and only Open WebUI reaches it over the internal Docker network.
• WEBUI_SECRET_KEY is the seed used to sign JWT session cookies. Generate a strong one with openssl rand -hex 32 before the first launch.

Pinned image tags matter here more than they might seem. A floating tag such as latest can change under the stack overnight, and a restart on a Tuesday could then pull a runtime nobody tested. With fixed versions, a rebuild next year gives the same result as today, and an upgrade becomes a deliberate edit to one line that can be reviewed and rolled back.

Starting it and pulling a model

Bring the stack up in the background:

docker compose up -d

Check container state and port bindings:

docker compose ps

Pull a first model by running the command inside the Ollama container:

docker compose exec ollama ollama pull llama3.2:8b

Confirm the model registered along with its quantization:

docker compose exec ollama ollama list

TLS in front of everything

Port 3000 should never face the public network without TLS and host routing. Caddy handles this well, since it manages HTTPS certificates automatically. Install it on the host and write /etc/caddy/Caddyfile:

ai.yourdomain.com {
    reverse_proxy 127.0.0.1:3000 {
        header_up Host {host}
        header_up X-Real-IP {remote_host}
    }
}

Reload Caddy so it fetches a certificate and starts proxying over an encrypted connection:

sudo systemctl reload caddy

Growing beyond chat

Teams eventually want autonomous multi-step agents and workflow automation, not only conversations. Open WebUI can then sit alongside orchestration frameworks like Dify, or alongside a high-throughput backend such as vLLM. Because Ollama offers a standard REST API on the internal network, other workflow tools on the same bridge can use its models directly.

Troubleshooting

These are recurring problems with their causes and fixes.

• WebUI shows a backend connection error. The cause is an incorrect OLLAMA_BASE_URL or a network mismatch between containers. Confirm the variable matches the internal DNS name, http://ollama:11434, then test reachability from inside the Open WebUI container by curling the /api/tags path on the Ollama host name.
• CUDA GPU isn't detected in Ollama. Either the NVIDIA Container Toolkit is missing or the compose file lacks a GPU reservation block. Install nvidia-container-toolkit on the host, run sudo systemctl restart docker, and check that reservations match the compose spec.
• Tokens per second drop sharply over time. The model is larger than available VRAM, so layers spill into system RAM. Look at allocation with nvidia-smi, then switch to a smaller model or a more compact quantization (Q4_K_M instead of Q8_0).
• Permission denied on the open_webui_data volume. Directory permissions on the host don't match the container process UID of 1000. Run sudo chown -R 1000:1000 /var/lib/docker/volumes/open_webui_data.
• WebSocket drops during streaming. The reverse proxy is buffering streaming responses or failing upgrade requests. Make sure it forwards the Upgrade and Connection headers cleanly with buffering off.

Backups

User accounts, chat history, RAG indexes and configuration have to be preserved. Model weights can always be downloaded again, so leave them out of backups to save space. Write a safe, non-destructive script at /usr/local/bin/backup-local-ai.sh. It begins with the interpreter line and strict mode:

#!/usr/bin/env bash
set -euo pipefail

The body sets a backup directory and timestamped file name, pauses Open WebUI, archives the data volume, and restarts the container:

BACKUP_DIR="/var/backups/local-ai"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
TARGET_FILE="${BACKUP_DIR}/open_webui_data_${TIMESTAMP}.tar.gz"mkdir -p "${BACKUP_DIR}"echo "Stopping Open WebUI container..."
docker stop open-webuiecho "Archiving Open WebUI database and user storage..."
tar -czf "${TARGET_FILE}" -C /var/lib/docker/volumes/open_webui_data/_data .echo "Restarting Open WebUI container..."
docker start open-webuiecho "Cleaning backups older than 14 days..."
find "${BACKUP_DIR}" -type f -name "*.tar.gz" -mtime +14 -deleteecho "Backup execution complete: ${TARGET_FILE}"

Make it executable and open the crontab:

sudo chmod +x /usr/local/bin/backup-local-ai.sh
sudo crontab -e

Then add a line to run it daily at 3:00 AM:

0 3 * * * /usr/local/bin/backup-local-ai.sh >> /var/log/local-ai-backup.log 2>&1

Final checklist

• Network isolation: Ollama's port 11434 is bound to loopback (127.0.0.1) and not reachable publicly.
• Authentication: WEBUI_AUTH=true is set explicitly in Open WebUI.
• Key integrity: a custom 64-character WEBUI_SECRET_KEY is configured.
• Image pinning: images use pinned versions, not floating tags.
• VRAM limits: keep-alive and concurrency caps are tuned to prevent GPU driver crashes.
• Backups: database volumes are archived daily with retention.

For reference, the primary documentation lives in the Ollama GitHub repository, the NVIDIA Container Toolkit docs and the Docker Compose specification.