Qwen3.8-27B on One RTX 5090: vLLM, LACT Fan Control and Concurrency Benchmarks

A single RTX 5090 with 32 GB can run Qwen3.8-27B as a private, OpenAI-compatible LLM server. This setup uses vLLM on headless Ubuntu Server 24.04, adds Prometheus and Grafana for monitoring, fixes a nasty thermal problem with LACT, and then measures what happens as concurrency climbs.

The aim isn't to chase a synthetic peak figure. It's to show what one card can realistically deliver as an internal server. The final configuration reaches about 177 output tokens per second for a single stream and 1,583 tokens per second in aggregate with 12 requests at once. Beyond that point, latency saturates in a clear way.

Everything from GPU passthrough and vLLM settings to thermal control, benchmarking, context management and network capacity is covered below.

Quick facts

• Hardware: an RTX 5090 with 32 GB, a Ryzen 9 9900X and 64 GB of RAM, running Qwen3.8-27B on headless Ubuntu Server 24.04.
• Serving stack: vLLM with mixed NVFP4/FP8 precision, an FP8 KV cache, prefix caching, chunked prefill and MTP speculative decoding.
• Thermals: under sustained inference the first configuration pushed the GPU to roughly 85°C. LACT brought fan control under a daemon and kept sustained temperatures around 60 to 70°C.
• Throughput: about 177 output tokens per second with one request, and about 1,583 aggregate at 12.
• Saturation: 16 concurrent requests delivered the same throughput (about 1,579 tokens per second), while mean time to first token rose from roughly 141 ms to 1,343 ms.
• Takeaway: for this workload, 12 concurrent requests is the practical knee between throughput and latency.

Getting there on a headless server took more effort than starting a Docker container. Sustained inference exposed a fan-control gap, LACT closed it, and the benchmarks then showed where the hardware tops out.

Hardware and base OS

The machine was built to sustain loads above 550 W without bottlenecks elsewhere in the system:

• CPU: AMD Ryzen 9 9900X, 12 cores and 24 threads
• RAM: 64 GB of DDR5
• GPU: NVIDIA GeForce RTX 5090, a Palit GameRock with 32 GB of GDDR7
• Storage: a 2 TB PCIe 4.0 NVMe drive
• OS: Ubuntu Server 24.04.3 LTS, purely headless with no desktop environment
• Driver and CUDA: NVIDIA driver 595.84 with CUDA 13.2

Check the OS release first:

lsb_release -a

Docker and the NVIDIA Container Toolkit

Start with a clean Docker runtime on the host, with GPU passthrough turned on.

Install Docker Engine

curl -fsSL https://get.docker.com | sh

Add the NVIDIA Container Toolkit repository

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \

sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \

sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Install the toolkit and restart Docker

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

docker -v

The last command should report Docker Engine 29.1.3, which is ready for GPU workloads.

Observability with Prometheus and Grafana

Set up metrics before starting inference. A 575 W TGP card under heavy load shouldn't run unobserved.

Version 1.14.0 of nvidia_gpu_exporter does the job. It's a single Go binary that pulls NVML counters and serves them as a Prometheus target on port 9835:

cd /opt

wget https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v1.14.0/nvidia_gpu_exporter_1.14.0_linux_x86_64.tar.gz

tar -xzf nvidia_gpu_exporter_1.14.0_linux_x86_64.tar.gz

Add a systemd unit so it keeps running:

# /etc/systemd/system/nvidia_gpu_exporter.service
[Unit]
Description=NVIDIA GPU Exporter
After=network.target

[Service]
Type=simple
ExecStart=/opt/nvidia_gpu_exporter
Restart=always

[Install]
WantedBy=multi-user.target

Then enable and check it:

sudo systemctl enable --now nvidia_gpu_exporter

sudo systemctl status nvidia_gpu_exporter

With Prometheus scraping the host, build a Grafana dashboard with core temperature, hotspot, fan speed in percent, power draw and VRAM use. That dashboard turns out to matter within minutes.

Deploying vLLM with Qwen3.8-27B NVFP4

The model is the Qwen3.8-27B-NVFP4-RTX5090 checkpoint, which is tuned for NVIDIA Blackwell GPUs. This is the production compose.yml:

services:
  vllm:
    image: vllm/vllm-openai:v0.27.1
    container_name: vllm-server
    restart: always

    runtime: nvidia
    ipc: host
    shm_size: 32g

    volumes:
      - ./hf_cache:/root/.cache/huggingface
      - ./hf_cache:/root/.cache/vllm

    environment:
      HF_TOKEN: my_hf_token
      HF_HUB_DISABLE_TELEMETRY: "1"
      HF_HUB_DISABLE_IMPLICIT_TOKEN: "1"
      # Target Blackwell architecture (RTX 5090 / Compute Capability 12.0)
      CUTE_DSL_ARCH: "sm_120"
      PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:  True"

    ports:
      - "8000:8000"

    networks:
      - rag-net

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

    command:
      - gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
      - --host
      - "0.0.0.0"
      - --port
      - "8000"
      - --tensor-parallel-size
      - "1"
      - --quantization
      - modelopt
      # FP8 KV cache allocation for memory efficiency
      - --kv-cache-dtype
      - fp8
      # 128k context window support
      - --max-model-len
      - "131072"
      - --max-num-seqs
      - "16"
      - --max-num-batched-tokens
      - "8192"
      # Allocate 93% of 32 GB VRAM to weights and KV cache pool
      - --gpu-memory-utilization
      - "0.93"
      - --enable-chunked-prefill
      - --enable-prefix-caching
      - --limit-mm-per-prompt
      - '{"image":10,"video":{"count":1,"num_frames":2}}'
      - --reasoning-parser
      - qwen3
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_xml
      - --trust-remote-code
      # Speculative decoding via Multi-Token Prediction (MTP)
      - --speculative-config
      - '{"method":"mtp","num_speculative_tokens":3}'
      - --served-model-name
      - llm-qwen3.8

networks:
  rag-net:
    driver: bridge

Start the service

Run the stack detached:

docker compose up -d

Watch the download and startup

A 27B model needs a few minutes on first boot. vLLM downloads the model files into ./hf_cache, loads the FP4 weights onto the card, compiles CUDA graphs for Blackwell (sm_120) and pre-allocates the FP8 KV cache pool. Follow the logs while it works:

docker logs -f vllm-server

The API server will register the OpenAI-compatible endpoints (/v1/chat/completions, /v1/models and /metrics). It's ready once the launcher says the HTTP server has started. The health endpoint confirms it too:

curl -i http://localhost:8000/health

Check VRAM allocation

nvidia-smi

After initialization, vLLM holds about 28.5 GB of VRAM, covering model weights plus the pre-allocated FP8 KV cache.

The surprise: a headless thermal trap

With vLLM running, a first batch of concurrent requests tested stability. The Grafana graphs showed a serious problem. Temperatures kept breaking through the 80°C red line, and the fans bounced between 50% and 80% instead of holding a steady speed. The core went past 80°C and peaked at 85°C, while memory junction temperatures ran even higher.

The cause lies in how NVIDIA handles consumer cards on Linux:

1. Desktop VBIOS defaults. The factory BIOS on the Palit GameRock favors quiet operation for desktop users, so the fans spin up late.
2. Headless limits. Normal fan control through nvidia-settings requires a running X11 server and the Coolbits option.
3. No command-line route. On a server without X11, nvidia-smi rejects manual fan commands for GeForce cards. Those controls are reserved for datacenter hardware such as the A100, H100 and L40.

During batch work the card pulled heavy power inside a closed chassis, and the standard NVIDIA tools had no way to demand faster fans.

Fixing it with LACT

The answer is LACT, the Linux AMD and NVIDIA Controller Toolkit. It runs as a system daemon (lactd) and talks to the hardware controllers through a Unix socket at /run/lactd.sock, so no X11 or display is involved.

1. Install the daemon

cd /tmp

wget https://github.com/ilya-zlobintsev/LACT/releases/download/v0.10.1/lact-0.10.1-0.amd64.ubuntu-2404.deb

sudo apt install ./lact-0.10.1-0.amd64.ubuntu-2404.deb netcat-openbsd -y

sudo systemctl enable --now lactd

2. Find the GPU

lact cli list-gpus

The output lists the card as 10DE:2B85-1569:F318-0000:01:00.0, a GeForce RTX 5090 marked Dedicated.

Test the socket

echo '{"command":"device_info","args":{"id":"10DE:2B85-1569:F318-0000:01:00.0"}}' \
  | ncat -U /run/lactd.sock \
  | grep -o '"flags":\[[^]]*\]'

The reply should contain "flags":["ConfigurableFanControl","AutoFanThreshold"], which confirms that curve control is available at the hardware level.

3. Apply a custom fan curve

The curve is aggressive and step-wise. The fans stay off below 40°C so an idle machine is quiet, then ramp hard when inference begins:

Temperature Fan Speed Target State
< 40°C 0% Fan-Stop (Idle)
 40°C 30% Initial ramp
 50°C 35% Low load
 55°C 40% Moderate load
 60°C 50% Sustained inference
 65°C 60% Active cooling
 70°C 70% High load
 75°C 80% Aggressive dissipation
 80°C 90% Thermal boundary
 85°C 100% Maximum cooling

Push it through the socket and confirm:

# 1. Apply curve configuration (generates a 5-second pending state)
echo '{"command":"set_gpu_config","args":{"id":"10DE:2B85-1569:F318-0000:01:00.0","config":{"fan_control_enabled":true,"fan_control_settings":{"mode":"curve","static_speed":0.4,"temperature_key":"edge","interval_ms":500,"curve":{"40":0.30,"50":0.35,"55":0.40,"60":0.50,"65":0.60,"70":0.70,"75":0.80,"80":0.90,"85":1.0},"spindown_delay_ms":3000,"change_threshold":2,"auto_threshold":40}}}}' | ncat -U /run/lactd.sock

# 2. Confirm the pending change immediately
echo '{"command":"confirm_pending_config","args":{"command":"confirm"}}' | ncat -U /run/lactd.sock

After confirmation, lactd writes the active configuration to /etc/lact/config.yaml, so the curve survives server reboots. You can check what was saved:

grep -A 20 -i "fan_control" /etc/lact/config.yaml

The effect on the dashboard was immediate. Under continuous inference, temperatures settled between 60°C and 70°C. With the heat under control, throughput testing could begin.

Benchmarking concurrency with vllm bench serve

To find practical serving limits, the test uses vLLM's own benchmark suite at several concurrency levels.

The test setup:

• Workload: 512 input tokens and 512 output tokens
• Volume: 200 requests per concurrency level
• Concurrency levels: 1, 2, 4, 8, 12 and 16
• Sampling: temperature 0, so results are deterministic

A script automates the runs. It steps through each level inside the container, waits 10 seconds between runs so the card stays thermally stable, and collects latency and speculative-decoding statistics:

#!/usr/bin/env bash
set -u
set -o pipefail

CONTAINER="vllm-server"
BASE_URL="http://localhost:8000"
MODEL="llm-qwen3.8"
TOKENIZER="/root/.cache/huggingface/hub/models--gittensor-model-hub--Qwen3.8-27B-NVFP4-RTX5090/snapshots/0cc27958cefbbe231782ec8511de8c4eb5233348"

INPUT_LEN=512
OUTPUT_LEN=512
NUM_PROMPTS=200
CONCURRENCIES=(1 2 4 8 12 16)
SLEEP_BETWEEN=10
SEED=42

for CONCURRENCY in "${CONCURRENCIES[@]}"; do
    docker exec "$CONTAINER" \
      vllm bench serve \
      --backend openai \
      --base-url "$BASE_URL" \
      --model "$MODEL" \
      --tokenizer "$TOKENIZER" \
      --dataset-name random \
      --random-input-len "$INPUT_LEN" \
      --random-output-len "$OUTPUT_LEN" \
      --num-prompts "$NUM_PROMPTS" \
      --max-concurrency "$CONCURRENCY" \
      --num-warmups 3 \
      --temperature 0 \
      --seed "$SEED" \
      --disable-tqdm
    sleep "$SLEEP_BETWEEN"
done

Results and the saturation point

This table summarizes the aggregate figures across the whole test matrix:

=============================================================================
CONC     AVG TOK/S    TTFT ms    TPOT ms    ITL ms     ACCEPT %    ACCEPT LEN
=============================================================================
1        176.86       93.17      5.48       19.63      83.37%      3.50
2        377.42       103.26     5.10       18.06      85.17%      3.56
4        697.96       106.06     5.50       19.63      86.03%      3.58
8        1213.43      119.17     6.28       22.42      85.99%      3.58
12       1582.98      141.35     7.16       25.46      85.47%      3.56
16       1579.04      1342.82    7.23       25.62      85.06%      3.55
=============================================================================

One stream and MTP

With a single request, the system reached about 177 output tokens per second and a time to first token of 93 ms. Multi-Token Prediction contributes to that decoding speed. The speculative acceptance rate stayed above 85%, with 3.55 tokens accepted per decoding step on average.

Saturation at 12 concurrent requests

Aggregate throughput grew almost in a straight line from 176 tokens per second up to 1,583 at 12 concurrent requests. Going from 12 to 16 shows where the limit sits:

• Throughput leveled off at 1,579 tokens per second.
• Mean time to first token jumped from 141 ms to 1,342 ms, a 9.5-fold rise.

At 16 requests, the compute cores are already busy generating tokens for batches in flight. New prompts wait in vLLM's prefill queue, which adds visible delay before their first token appears.

What 12 slots mean for real users

It's easy to assume that a limit of 12 concurrent requests means only 12 employees can use the server. That's a misreading. Concurrency counts requests the inference engine handles at the same time, sharing GPU compute across overlapping scheduling windows.

Take an internal office workload such as documentation search, internal chat or coding assistants:

• A person sends a prompt and reads the answer.
• Reading, thinking and typing take 15 to 45 seconds before the next message.
• A typical 512-token reply at 120 to 170 tokens per second finishes in about 3 to 4 seconds.

If each active session uses the GPU for roughly 3 seconds out of every 30, a single slot can cover around 10 chat sessions per minute. Under that simple duty-cycle estimate, 12 slots support far more people than the raw number suggests. In practice the safe user count depends on actual request patterns, prompt size, output length, RAG workload and latency requirements.

Managing context length for long chats and RAG

The compose file sets max-model-len 131072, a 128K-token context. The model supports much longer windows, but using the full length on a 32 GB card would raise memory pressure sharply and cut serving efficiency.

Each extra token in active context takes space in the KV cache and adds to prefill time. In heavy RAG or multi-turn agent pipelines:

• Don't dump whole histories. Long conversation logs add up in memory and slow time to first token for every concurrent user.
• Use sliding windows and summaries. Keep recent turns word for word, condense older exchanges into structured summaries, and drop irrelevant document chunks once they've been ingested.
• Use prefix caching. With the enable-prefix-caching flag, vLLM computes repeated system prompts and standard company instructions once and keeps them cached, so team requests skip redundant prefill work.

What this setup proves

Running a capable LLM locally is no longer only a proof of concept. One RTX 5090 can act as a practical private inference server for internal applications, provided the software stack and workload are set up carefully.

In this deployment, vLLM, mixed NVFP4/FP8 quantization, MTP speculative decoding, prefix caching and GPU monitoring together delivered about 177 output tokens per second for a single stream and 1,583 in aggregate at 12 concurrent requests.

The benchmark also showed why maximum concurrency isn't the same as the best concurrency. Raising the limit from 12 to 16 added almost no throughput and increased mean time to first token by roughly 9.5 times.

An on-premise server brings several further benefits for internal workloads:

• Privacy and control. Proprietary code, documents and internal knowledge never go to an outside inference API.
• Predictable cost. A fixed hardware bill replaces per-token billing, which matters more as utilization rises.
• Low latency. A local API serves internal chat, RAG, coding assistants and automation without an external round trip.
• Full control of the stack. Quantization, caching, concurrency, context length, thermals and monitoring can all be tuned to the real workload.
• Shared capacity. One GPU can serve several applications and users, within the limits the benchmark shows.

The lesson isn't that every RTX 5090 will reach these exact numbers. It's that a carefully configured consumer GPU can become a surprisingly capable private LLM server reachable over the network. The configuration, benchmark and thermal tuning above make a practical starting point for building one.