Running the Same OpenAI Chat Loop Locally with Ollama

Chat loops against hosted providers cost money, and every prompt travels to someone else's data center. Running an open model on your own machine with Ollama removes both. The best part is that the existing OpenAI-based chat code keeps working, because only the address changes.

Same dialect, different machine

Ollama is a local model server. After installing it, you pull a model once, and the weights download the way a Docker image does. The server then listens for HTTP requests on localhost:11434. What makes this convenient is that Ollama deliberately mirrors OpenAI's Chat Completions API: the same /v1/chat/completions route, the same list of role-tagged messages, the same response shape with choices[0].message.content, and the same usage counters.

Moving an OpenAI chat loop over therefore isn't a rewrite. It's a change of address. The trade is plain to see. There's no API key, no per-token bill and no data leaving the machine, and in return you get the speed of your own hardware and the quality of an open model instead of a frontier one. For learning how agents work, where hundreds of iterations are normal, that's a very good deal.

Setup and a single turn

Grab Ollama from ollama.com. It installs in one step and runs the server in the background. Next, pull a small model. llama3.2, around 2 GB, is a sensible default. Nothing new is needed on the Python side, since the standard openai package simply points somewhere else:

from openai import OpenAI

# Part 2's SDK, different address
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required, unused
)

resp = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {
            "role": "system",
            "content": "You are a concise "
                       "assistant.",
        },
        {
            "role": "user",
            "content": "Where are you running "
                       "right now?",
        },
    ],
    temperature=0,
)
print(resp.choices[0].message.content)

Two settings deserve a note. Pointing base_url at localhost reroutes all traffic away from OpenAI. The api_key="ollama" string looks pointless but isn't: the SDK refuses to create a client with no key at all, while Ollama itself ignores whatever you send, so any non-empty text would do. Finally, model must be a name you've pulled already. There's no pricing tier to select, only what sits on your disk, which ollama list will print.

The history loop stays the same

A good test of "same dialect" is whether the earlier ask() function runs against Ollama without any edit. It does. Only the code that creates the client is different.

A short conversation proves it, starting with a system message and growing from there:

msgs = [{
    "role": "system",
    "content": "You are a concise assistant.",
}]

def ask(text: str) -> str:
    msgs.append(
        {"role": "user", "content": text}
    )
    resp = client.chat.completions.create(
        model="llama3.2",
        messages=msgs,  # full history
        temperature=0,
    )
    reply = resp.choices[0].message.content
    msgs.append({
        "role": "assistant",
        "content": reply,
    })
    return reply

print(ask("Define a context window."))
print(ask("Now for a five-year-old."))
print(ask("Which answer was shorter?"))

The third question still demonstrates the history mechanism, and everything learned from the hosted version carries over. The server remembers nothing between calls. The growing msgs list is the reason later turns take longer. You aren't billed here, but you do pay in seconds, and watching a local model slow down as history accumulates makes context cost feel real in a way an invoice never could.

Set expectations accordingly. A model with around 3 billion parameters is noticeably less sharp than GPT-4o or Claude. It may ramble past the system prompt or trip over a comparison such as which of two things was shorter. That isn't a bug in the loop. It's what three billion parameters buys, and seeing the gap firsthand teaches more than reading benchmarks.

Running the example

Three commands cover it. The first installs the model, the second installs Python dependencies, and the third runs the script:

# one-time: install from ollama.com, then
ollama pull llama3.2

pip install -r requirements.txt
python examples/part05_ollama.py

Before anything else, the script verifies that the Ollama server answers on localhost:11434. If it doesn't, the script exits with instructions for installing and pulling. That's the local counterpart of failing fast on a missing API key. It then runs the single turn and the three-turn conversation and logs token usage, because Ollama reports usage in the same fields. There's no key and no cost. It also stays out of CI, for a different reason than the hosted examples: CI machines don't run a model server.

When something goes wrong

• A "model not found" error means that exact tag hasn't been pulled yet, so run ollama pull llama3.2.
• A connection error means the server isn't running. Start it with ollama serve, although desktop installs launch it automatically.
• A pause of several seconds before the first token on first use is the model being loaded into memory. Nothing has hung.

Why a local endpoint pays off

Agent experiments repeat endlessly. Retrieval pipelines get rebuilt, reasoning loops fire several model calls for one question, and graph workflows are rerun after every small debugging tweak. Paying per token for each of those runs adds up, while a free local endpoint lets you experiment without watching a meter. Since Ollama speaks the OpenAI dialect, any example can run either way by swapping the base URL and the model name, nothing else. The result is a sandbox with unlimited tokens.

The obvious next topic is RAG. A model can't know your documents, so the real question becomes which tokens earn a place in the context window.