The Stateless Chat Loop: Calling the OpenAI API by Hand in Python

Frameworks like LangChain make chat models feel like objects with memory, but the API underneath remembers nothing. Every call is independent, and the "conversation" is a list of messages your code rebuilds and resends each time. Writing that loop by hand once with the OpenAI Python SDK shows exactly what agent frameworks automate, why token costs climb during a conversation and which errors to plan for.

What a hosted model endpoint really is

The OpenAI API is a simple arrangement: the provider runs the model on its GPUs and exposes inference over HTTPS. You send text, the model generates tokens in its usual next-token loop, and you pay per token in both directions. Three consequences follow:

1. It is stateless. Nothing from earlier requests is kept, so each request must contain everything the model should know.
2. It is plain HTTP. The SDK wraps a POST request, so when something fails you can inspect the raw traffic.
3. You buy tokens, not answers. A wordy prompt costs money on every call that includes it.

Never put the API key in source code. Keys leak through Git history, screenshots and shared notebooks, and a leaked key means someone else spends on your account. The SDK reads OPENAI_API_KEY from the environment automatically, so your script needs no key-handling code at all. API usage is billed separately from a ChatGPT subscription, and new accounts usually need a small prepaid balance.

Roles: the shared message format

A request carries a list of messages, each with a role:

• system holds your instructions, which the model weighs more heavily.
• user holds what the person wrote.
• assistant holds the model's earlier replies and, in agents, its tool calls.

This format is shared across the ecosystem. Claude and Gemini use the same idea with small differences, Ollama imitates it, and LangChain's SystemMessage, HumanMessage and AIMessage are these roles as classes. The provider flattens the list into one token sequence before generating, so roles are really structured prompt engineering.

A single request

The first example creates a client that reads the key from the environment, sends a system instruction plus one question, and prints the reply along with prompt and completion token counts from usage:

from openai import OpenAI

client = OpenAI()  # key from env

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": "You are a concise "
                       "Python assistant.",
        },
        {
            "role": "user",
            "content": "Why resend the whole "
                       "chat history each call?",
        },
    ],
    temperature=0,
)
print(resp.choices[0].message.content)
u = resp.usage
print(u.prompt_tokens, u.completion_tokens)

gpt-4o-mini is a cheap model suited to learning; moving to a larger one is a one-string change, though model names and prices change, so check the current list. temperature=0 minimizes sampling randomness, the right default for question answering and later for tool-calling agents. Log usage on every call; it is your cost meter.

Holding a conversation yourself

Since the server forgets everything, your code owns the history: after each call it stores the reply, adds the following question and resubmits everything. The helper below does this with a module-level msgs list that starts with a system message:

msgs = [{
    "role": "system",
    "content": "You are a concise assistant.",
}]

def ask(text: str) -> str:
    msgs.append(
        {"role": "user", "content": text}
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=msgs,  # full history
        temperature=0,
    )
    reply = resp.choices[0].message.content
    msgs.append({
        "role": "assistant",
        "content": reply,
    })
    return reply

print(ask("Define a context window."))
print(ask("Now for a five-year-old."))
print(ask("Which answer was shorter?"))

The third question proves the point. The model can only compare the two answers because both are in msgs and get resent. Drop the line that appends the assistant reply and it has no idea what you mean.

This ask() function reappears in many disguises. The ChatGPT web app is essentially it with a user interface. LangChain's RunnableWithMessageHistory is a managed version of append and resend. An agent's inner loop is the same pattern with tool calls and results added. Note the cost: turn three resends turns one and two, so input tokens grow with every exchange.

Running the example

Install the dependencies, export the key in your shell and run the script. The key shown is a placeholder; provide yours through the shell or a secrets manager, never a committed file:

pip install -r requirements.txt
export OPENAI_API_KEY="sk-..."
python examples/part02_chat.py

The script makes one single-turn call, then runs the three-turn conversation, and after each request it reports token usage and an approximate cost. It stops immediately if the key is missing and is kept out of CI on purpose, because it spends real money and needs a real key. If you want tests around such code, mock the client.

Pitfalls to design for early

1. Missing key: an AuthenticationError with HTTP 401, usually because the variable is not set in this shell, is misspelled or contains pasted whitespace. Check at startup and fail fast.
2. Rate limits: a RateLimitError with HTTP 429 means too many requests or an empty prepaid balance. Looping agents will hit it, so add retries with backoff now.
3. History fails both ways. Forget to append and the model forgets everything; append forever and token spend grows quadratically until you hit a context-length error. Real systems trim, summarize or retrieve only relevant history, and that last idea is the heart of RAG.
4. Temperature 0 makes output stable, not correct. Validate anything that matters.

The same shape across providers

Claude takes a list of user and assistant messages, with the system prompt moved to a separate top-level parameter. Gemini uses the same conversation-as-list model, with roles named user and model. Ollama offers an OpenAI-compatible endpoint, so this code can target a local model by changing the base URL and model name; see calling Claude, GPT and Gemini through OpenAI-compatible endpoints. That convergence is what lets LangChain offer one abstraction over many providers.

Key takeaways

• The chat API is stateless; your code owns and resends the conversation.
• The role-tagged message list is effectively a cross-provider standard.
• Log usage on every call, since input tokens grow each turn.
• Keep keys in the environment and live API calls out of CI.
• Handle 401s, 429s and unbounded history before building agents, because agent loops amplify all three.