Fixing Ollama Connection Refused on WSL2 and Wiring Up Dify and vLLM

Ollama ran fine on a Windows 11 machine. You could pull a model and chat with it straight from the WSL2 terminal. The moment another service tried to talk to it, though, the connection was refused. The models weren't the issue. The network was.

What follows is the path from that broken, isolated setup to one that survives reboots and serves local models across several virtual boundaries.

The goal, and where it fell over

The aim was a local AI development stack for people who want to work with LLMs without sending private data to a cloud provider. Ollama would cover quick experiments and local RAG workflows. vLLM would handle heavier, higher-throughput inference. Both would then plug into Dify, an open-source LLM app builder running in Docker.

The first steps went smoothly. Ollama was installed inside a WSL2 Ubuntu distribution, the llama3 model was pulled, the CLI worked, and a test request to the local API from the same Linux shell came back instantly.

Then Dify, running in a Docker container on the host network, reported a connection failure. The first suspect was Dify's configuration, and that suspicion was wrong. The real cause was a misunderstanding of how a service binds to a network interface, made worse by the separate network namespaces that WSL2 and Docker each create.

Think of the layout as nested rooms. The Windows 11 host sits outside. Docker Engine forms one namespace, with the Dify API and Dify worker containers inside it. The WSL2 utility VM forms another, holding Ollama on port 11434 and vLLM on port 8000. Two boundaries matter: the one between Windows and the WSL2 VM, and the one between the Docker network and the host. The infamous localhost confusion lives at those walls.

Running doesn't mean reachable

A healthy service isn't necessarily a reachable one. Ollama was alive, but that only meant its process was up and bound to one particular interface deep in the Linux kernel.

A server doesn't listen to "the network" in the abstract. It binds to a specific IP address and port, and a client gets through only by sending packets to a routable address that matches that binding. Many safe defaults bind to 127.0.0.1, the IPv4 loopback, which confines the service to the very namespace it started in. Binding to 0.0.0.0 makes it answerable on every interface of the machine, and that is mandatory if WSL2 has to be bridged to Windows or Docker.

Finding the refused connection

When something can't reach Ollama, start by working out which layer is failing: DNS, TCP, or the application. Here the symptom was Dify reporting a connection error. The hypothesis was a failed TCP handshake across the VM boundary. The test was a raw TCP check from the Windows side, which is a different namespace from WSL2 and skips any application logic. From PowerShell:

Test-NetConnection -ComputerName localhost -Port 11434

It returned TcpTestSucceeded : False, so the handshake was failing and Dify wasn't to blame. Next, look at the listening socket from inside Linux, in the WSL2 terminal:

ss -tlnp | grep 11434

The output settled it:

LISTEN 0  4096  127.0.0.1:11434  0.0.0.0:*  users:((ollama,pid=1234,fd=7))

According to the ss listing, Ollama was listening only on 127.0.0.1:11434. Anything arriving over the WSL2 virtual Ethernet interface, which is what Windows uses to talk to the VM, got rejected by the kernel on the spot.

Three terms worth getting straight

localhost is just a hostname that gets resolved through the hosts file. Which machine it actually points at depends on the container or VM you happen to be typing it in.

127.0.0.1 is the IPv4 loopback address. It's a virtual interface that lives inside a single OS kernel, and packets sent to it never reach a physical or virtual network card.

Then there's the wildcard, 0.0.0.0 (INADDR_ANY). When a server binds to it, the OS accepts incoming TCP connections on every IPv4 interface it has.

Microsoft's WSL networking documentation at https://learn.microsoft.com/en-us/windows/wsl/networking explains that the hypervisor runs a lightweight utility VM with its own network adapter and NAT address, so Windows has to cross that NAT to reach the Linux side.

The one change that fixes it

Make Ollama bind to all interfaces by setting the OLLAMA_HOST environment variable. If systemd manages the process, don't modify the main unit file, because package updates will overwrite your edit. Create a drop-in override instead. Inside WSL2, first make the directory:

# Create the override directory
sudo mkdir -p /etc/systemd/system/ollama.service.d

Then write the override file:

# Create and edit the configuration file safely
echo '[Service]' | sudo tee /etc/systemd/system/ollama.service.d/override.conf
echo 'Environment=OLLAMA_HOST=0.0.0.0' | sudo tee -a /etc/systemd/system/ollama.service.d/override.conf# Reload systemd and restart the daemon
sudo systemctl daemon-reload
sudo systemctl restart ollama

After a restart, check the socket again. This is what success looks like:

LISTEN 0  4096  *:11434  *:*  users:((ollama,pid=1234,fd=7))

The asterisk means it's listening on all interfaces, and repeating the network test from Windows now gave True.

Why pasting the WSL2 IP is a trap

At first, Dify was pointed at Ollama by running ip addr show eth0 in WSL2, copying the address and pasting it into the Dify settings, something like http://172.20.5.42:11434. It worked for a short while and is a brittle design. The Hyper-V virtual switch hands out that address dynamically, so it changes after every reboot or service restart and quietly breaks the integration.

A sturdier approach lets the Windows host forward traffic deterministically with netsh portproxy. To make it survive reboots without manual work, use a PowerShell script run as Administrator on the host. The first part looks up the current NAT address of the WSL2 instance:

# Dynamically fetch the current WSL2 NAT IP
$wsl_ip = (wsl -d Ubuntu -e sh -c 'ip -4 addr show eth0 | grep -oP ''(?<=inet\s)\d+(\.\d+){3}''')

The second part removes stale routing rules and adds fresh ones that point at the current address:

# Clear existing stale routing rules
netsh interface portproxy delete v4tov4 listenport=11434 listenaddress=0.0.0.0
netsh interface portproxy delete v4tov4 listenport=8000 listenaddress=0.0.0.0# Add the new dynamic rules for Ollama (11434) and vLLM (8000)
netsh interface portproxy add v4tov4 listenport=11434 listenaddress=0.0.0.0 connectport=11434 connectaddress=$wsl_ip
netsh interface portproxy add v4tov4 listenport=8000 listenaddress=0.0.0.0 connectport=8000 connectaddress=$wsl_ipWrite-Host 'Port forwarding established to WSL2 IP: ' $wsl_ip

The effect is that localhost requests on Windows land on whatever IP the Linux instance currently has. Save the script and have Windows Task Scheduler run it at startup, and manual IP bookkeeping goes away.

Bringing in vLLM

With Ollama stable, vLLM came next. Its PagedAttention design gives a big throughput edge when many generations run concurrently. The server was started inside WSL2 with the host flag set, the port exposed and GPU memory allocated conservatively to avoid out-of-memory crashes:

python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --host 0.0.0.0 \
  --port 8000 \
  --gpu-memory-utilization 0.8

Before opening the Dify UI, check the API route on its own. Mixing up application errors with network errors can burn hours. JSON payloads need double quotes, which makes quick bash one-liners awkward, so a short Python script does the job. It starts with the import:

import requests

and continues with the request itself:

api_url = 'http://localhost:8000/v1/chat/completions'
headers = {'Content-Type': 'application/json'}
payload = {
    'model': 'mistralai/Mistral-7B-Instruct-v0.3',
    'messages': [{'role': 'user', 'content': 'Explain network routing briefly.'}]
}response = requests.post(api_url, headers=headers, json=payload)
print(response.status_code)
print(response.json())

An HTTP 200 means vLLM is working as an OpenAI-compatible server. If that call had failed, the problem would be known to sit entirely inside the Linux VM.

The Docker localhost trap

Last obstacle: inside a container, localhost refers to the container itself. Entering http://localhost:8000/v1 in Dify makes the Dify API container look for a model within its own namespace, and it fails immediately.

Since the Windows portproxy already exposes the AI services on the host, the container only needs a way out to the host. Docker Desktop provides a special DNS name for that, described at https://docs.docker.com/desktop/networking/. In Dify, open Model Provider, choose OpenAI-API-compatible and fill in:

• Endpoint: http://host.docker.internal:8000/v1
• Model: whatever vLLM serves, here mistralai/Mistral-7B-Instruct-v0.3
• Key: the literal word EMPTY, because local instances don't check it

The setup worth keeping

The final flow looks like this. A request leaves the Dify container for host.docker.internal:11434. The Windows host receives it and the netsh portproxy rule fires. The rule forwards it to the 172.x.x.x address of the WSL2 virtual NAT, where the Ollama daemon, bound to 0.0.0.0, accepts the payload.

That setup ends manual IP handling, and the bridge between the host OS, the hypervisor and the container engine now behaves the same way every day. Knowing how these namespaces fit together matters for more than fixing a flaky connection; it's the base skill for running dependable local AI infrastructure.