From Next-Token Prediction to Working Code: How LLMs Generate Python

Tools like ChatGPT and Copilot write functions, fix bugs and explain algorithms, and it can seem like something more than text completion is going on. The mental model below starts at the bottom: token prediction, attention, and the reason the mechanism that finishes a sentence also produces valid Python.

One question, asked repeatedly

Whether the output is a haiku, a marketing email or a merge sort, a language model is answering a single question over and over: given everything seen so far, what is the most likely next token?

There is no semantic understanding in the traditional sense, and no stage at which the model resolves to write a function ahead of producing the word def. It predicts token by token. With billions of parameters and training text counted in the hundreds of billions of tokens, the predictions end up looking much like reasoning.

What a token is

Precision helps before getting into prediction. Models don't work on characters or words but on tokens, subword units produced by an algorithm such as Byte Pair Encoding. Roughly:

• Common words usually become single tokens, such as "the", "def" and "return".
• Rare words split into pieces, so "unhappiness" becomes ["un", "happiness"].
• Code symbols are often single tokens, for instance "(", ":", "==" and "->".

For GPT-4, one token is about 0.75 English words, and in code it is closer to 3 or 4 characters, because code is full of one-character tokens like : and (.

The model also has a vocabulary, a fixed list of known tokens that typically runs from 50,000 to 100,000 entries. At each step it assigns a probability to every entry and either picks the likeliest or samples from the distribution, with the choice governed by temperature.

The prediction loop

For the sequence "The cat sat on the", the quantity to compute is a conditional probability:

P(next_token | "The", "cat", "sat", "on", "the")

That is a distribution over the whole vocabulary. A forward pass through the network produces a vector the size of the vocabulary, which a softmax function turns into a valid probability distribution:

P(token_i) = exp(logit_i) / Σⱼ exp(logit_j)

The highest-probability token is chosen in greedy decoding. Otherwise a token is sampled in proportion to the distribution, with temperature controlling the spread:

• temperature = 0 always takes the top token, so output is deterministic.
• temperature = 1 samples from the raw distribution, which is varied and creative.
• Above 1 the distribution flattens and output gets more random.
• Below 1 it sharpens and output becomes more conservative.

This is why one prompt can produce different answers on different runs.

The Transformer

The function that maps a token sequence to a next-token distribution is a Transformer, specifically a decoder-only one, which is the design behind GPT, Llama, Mistral and Claude. The high-level flow:

Token IDs → Embedding layer → [Attention + FFN] × N layers → Linear head → Softmax → P(next token)

The pieces, in order:

1. Embedding layer. Each integer token ID becomes a dense vector of size d_model, for example 4096 in Llama-3 8B. Positional embeddings, or RoPE in modern models, add information about order.
2. Transformer layers, repeated N times. Each holds multi-head self-attention, a feed-forward network, layer normalisation applied before each sub-layer in the "pre-norm" style, and residual connections.
3. Linear head and softmax. The final hidden state maps to logits of shape [|V|], and softmax produces the distribution.

Self-attention

Self-attention lets a token look at other tokens in the sequence to build a richer picture of itself. It is what gives Transformers an edge over RNNs: every token can reach every other token in one step, with no gradient fading over distance.

Given n tokens, every one of them gets projected three ways:

Q = XWᴬ_Q    (Query: what am I looking for?)
K = XWᴬ_K    (Key: what do I contain?)
V = XWᴬ_V    (Value: what do I contribute?)

Here X is the input matrix of shape [n, d_model] and the three W matrices are learned weights. The score between token i, acting as query, and token j, acting as key, is:

score(i, j) = qᵢ · kⱼᵀ / √dₖ

Dividing by the square root of the key dimension keeps dot products from growing too large when dₖ is high, which would push softmax into regions with tiny gradients. The full computation is:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

In a decoder-only model like GPT the attention is causal, or masked: token i may attend only to positions up to i. Future positions get a score of negative infinity before softmax, so their weights are exactly zero.

Multi-head attention runs h independent attention functions in parallel, each with the smaller dimension d_model / h, and concatenates the outputs:

MultiHead(Q, K, V) = Concat(head₁, ..., headₕ) · W_O
headᵢ = Attention(QWᵢ_Q, KWᵢ_K, VWᵢ_V)

Different heads end up attending to different relationships. One may follow syntactic dependencies, another coreference and another positional closeness.

The feed-forward network

After attention, each token's representation passes through a position-wise FFN:

FFN(x) = max(0, xW₁ + b₁)W₂ + b₂

Modern architectures such as Llama and Mistral use SwiGLU instead:

FFN(x) = (SiLU(xW_gate) ⊙ xW_up) · W_down

Most of a model's memorised knowledge sits in the FFN. It works like a key-value store over factual associations. Attention routes information between tokens, and the FFN looks up stored facts and patterns. In large models the FFN is about four times as wide as d_model. Llama-3 70B has d_model = 8192 and an intermediate size of 28672, and that is where most parameters live.

Why models can write code

Code is a token sequence just like natural language. To a model, a Python function is nothing special, only a run of tokens such as def, sort, _list, (, arr, ), : and so on.

Syntactically valid output is purely a consequence of the training distribution. Consider these conditional probabilities:

P("(" | ..., "def", " quicksort") ≈ 1.0
P(":" | ..., "def", " quicksort", "(", "arr", ")") ≈ 1.0
P("return" | context inside function body) >> P("import" | same context)

They are sharply peaked because the model has seen millions of Python files that all obey the same rules. Python's grammar isn't hard-coded anywhere in the network. It emerges from the statistics of the corpus.

It also explains why models sometimes write code that is perfectly formed yet logically wrong. They maximise the likelihood of the next token given the training distribution. They don't run the code and check it.

A sorting function, step by step

Suppose the prompt is "Write a Python function to sort a list". The model treats it as a token sequence and appends output one token at a time. Roughly:

• Step 1, after the prompt: def at about 0.91.
• Step 2, after def: sort at about 0.78.
• Step 3: _list or ( at about 0.65.
• Step 4, after sort(: arr or lst at about 0.71.
• Steps 5 to 7: ), then :, then a newline, each around 0.99.
• Step 8: the four-space indent at about 0.97.

In steps 5 to 8 the model approaches certainty on syntactic tokens. That isn't understanding Python. What happens is plain pattern completion, learned from a corpus in which every function definition looks the same.

The real choices come earlier, in steps 1 to 4, where the function name, parameter name and overall structure get chosen. Those choices are shaped by the prompt's content through attention. The token sort in the prompt carries high attention weight when the function name is being generated.

Temperature in code generation

Copilot and similar tools generally run at low temperature, around 0.1 to 0.3, for code:

• Syntactic tokens need to be nearly deterministic.
• There is often one correct implementation.
• Invented APIs become a major failure mode at high temperature.

For creative parts such as variable names, docstrings and comments, slightly higher temperature is acceptable. Some tools also use beam search, which keeps the top-k partial sequences alive at once and picks the one with the best joint probability at the end.

Why bigger behaves differently

A 7B model and a 70B model share the same architecture. The difference is scale: more layers, a wider d_model and more attention heads. Yet the larger one shows qualitative gains, including multi-step reasoning, more correct code and emergent in-context learning.

At scale the model learns more than surface token statistics. It builds compressed representations of deeper patterns such as algorithmic structure, logical dependencies and type constraints. A 70B model has seen enough buggy code next to its test failures that it has implicitly tied certain patterns to correctness. This is the empirical basis of the scaling hypothesis: capability follows a smooth curve in compute, data and parameters, and code quality follows it as reliably as any other skill.

Limits that follow from the design

Knowing the mechanism exposes some hard limits:

1. Execution. There is no runtime inside the model. What it emits resembles the output of a correct run, but nothing was actually run.
2. Guaranteed correctness. Prediction maximises likelihood over the training distribution, not logical validity.
3. True novelty. An algorithm with no statistical precedent in the training data can't be invented. The model can combine and adapt, but not originate.
4. Exact recall. There is no lookup table. All memory is spread across the weights, so APIs, function signatures and library versions will sometimes be hallucinated.

None of these are bugs awaiting a fix. They follow directly from the architecture, and knowing them makes anyone a much better user of coding assistants.

The whole picture in brief

• A token is a subword unit, about 0.75 English words.
• Prediction is a softmax over the vocabulary at each step.
• Self-attention lets each token attend to all earlier tokens through scaled query-key products.
• Multi-head attention runs several of those in parallel to capture different relations.
• The FFN acts as a key-value memory of factual associations.
• Code generation works like text generation, with syntax emerging from training statistics.
• Temperature sets how sharp the sampling distribution is.
• The hard limits are no execution, no guaranteed correctness and no true novelty.

Everything in the machinery, from attention and the FFN to residuals and layer norm, serves one purpose: making the conditional probability of the next token, given all prior tokens, as accurate as possible. The rest is emergent.