The Next-Token Loop: A Mental Model of LLMs Before You Touch Agents

Engineers who meet LLMs through a framework such as LangChain often cannot tell, when something breaks, whether the framework, the API it wraps, or the model itself is at fault. The cure is a clear mental model: an LLM is a next-token prediction function called repeatedly in a loop. With that picture in place, RAG is a way of choosing what goes into the function, ReAct a way of parsing what comes out, and LangGraph a way of orchestrating many calls. The examples below run locally with Python and one small library, no API key needed.

Generation is next-token prediction in a loop

Given a sequence of tokens, the model returns a probability for every possible next token. One is chosen, appended, and the longer sequence goes back in. This repeats until a stop token or a length limit. Streaming output simply exposes this loop token by token.

Two consequences follow:

• There is no plan. Each token depends only on the tokens before it.
• There is no memory between calls. Every request starts from exactly the tokens you send, which is the reason retrieval-augmented generation exists.

Tokens are subword pieces, not words

A model sees integers, not text. A tokenizer maps strings to token IDs and back. Frequent words usually become one token, while rare words and code are split into several. A handy estimate: one token is about three quarters of a word.

The snippet loads o200k_base, the encoding of the gpt-4o family, via tiktoken, encodes a sentence and prints each ID with its text fragment. Notice that "LangGraph" becomes two tokens and that leading spaces belong to the following token.

import tiktoken

# tokenizer of the gpt-4o model family
enc = tiktoken.get_encoding("o200k_base")

ids = enc.encode("LangGraph orchestrates agents.")
print(ids)
# [30741, 9922, 109873, 1381, 19297, 13]

for i in ids:
    print(i, repr(enc.decode([i])))
# 'Lang' 'Graph' ' orchestr' 'ates' ' agents' '.'

Training and inference are separate phases

Training adjusts the weights by predicting next tokens across large amounts of text. Inference runs the frozen model forward. Your prompt never changes the weights, so information the model lacks must be placed in the prompt at inference time. That is exactly what RAG does.

The context window is the model's entire world

A single call can process only so many tokens, and that limit, the context window, covers the prompt and the generated output together. Whatever is outside it does not exist for that call. Chat feels continuous only because the application re-sends earlier messages each time, and RAG exists because document collections do not fit, so a few relevant chunks are retrieved instead.

Budgeting is simple arithmetic. The helper counts prompt tokens with the tokenizer from above and subtracts them, plus a reserve for the answer, from a 128,000-token window, the figure given for gpt-4o-mini. Limits differ between models and versions, so check your provider's current documentation.

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

prompt = "Summarize the attached design doc."
window = 128_000     # e.g. gpt-4o-mini
reserve = 1_000      # room for the answer
used = count_tokens(prompt)
print("left:", window - reserve - used)

Temperature controls how the next token is picked

Sampling selects one token from the model's distribution, and temperature reshapes that distribution. Near zero, the top token almost always wins. At 1.0 and above, unlikely tokens get a real chance. Use temperature 0 for extraction, structured output and tool-calling agents, and roughly 0.7 to 1.0 for drafting and brainstorming. Our guide to temperature, top-k and top-p covers the other sampling knobs.

Building the loop in miniature

The next example pairs a real tokenizer with a fake model made from a lookup table. It is tiny, but it follows the same cycle as a production LLM: encode, predict, append, stop.

First, a small corpus is tokenized and each adjacent pair of IDs is counted. For every token, the table keeps only its most frequent successor, which makes this a greedy predictor, the equivalent of temperature zero.

CORPUS = (
    "the agent calls the model. "
    "the model returns a token. "
    "the agent calls the tool. "
    "the tool returns a result."
)

ids = enc.encode(CORPUS)
counts = {}
for a, b in zip(ids, ids[1:]):
    counts.setdefault(a, {})
    counts[a][b] = counts[a].get(b, 0) + 1

# greedy "model": token -> likeliest successor
table = {
    a: max(s, key=s.get)
    for a, s in counts.items()
}

The loop encodes a prompt, looks up the likeliest successor of the last token, appends it and prints the decoded text. A missing successor returns None, standing in for a stop token. The append is the key line: output becomes input.

seq = enc.encode("the agent calls")
for _ in range(3):
    nxt = table.get(seq[-1])
    if nxt is None:    # our toy "stop token"
        break
    seq.append(nxt)    # output becomes input
    print(enc.decode(seq))

The sequence grows one token per step:

the agent calls the
the agent calls the model
the agent calls the model.

One detail is instructive. In the corpus, " the" is followed by " model" and " tool" equally often, and max keeps whichever it saw first. Real models face such near-ties constantly, which is why sampling settings matter.

Running the examples

Collect the snippets in one script, for example examples/part01_tokens.py, with a requirements.txt that lists tiktoken. It prints token splits, does the budget arithmetic and runs the toy loop in about a second, offline once the tokenizer data is cached. Then change the corpus and watch the output change.

pip install -r requirements.txt
python examples/part01_tokens.py

Key takeaways

• An LLM is a frozen function from a token sequence to a next-token distribution, run in a feedback loop.
• It knows only its weights plus the context window and never learns from your prompt.
• Variety comes from sampling, controlled by temperature.
• Agent techniques decide which tokens go in and what happens to the tokens that come out.