From Next-Token Prediction to Instruction-Following: Fine-Tuning a GPT Model

Pretraining already gives a language model the ability to write. What it lacks is any sense of what a user wants done. Instruction tuning supplies that. Hand two models the same request, say to explain recursion to a ten-year-old, and the difference is obvious. The base model tends to echo the request and wander on into a textbook-flavored lead-in about computer science courses. It isn't answering at all, only extending the text. The tuned model offers a picture instead: a box holding a smaller box holding a smaller box, with recursion described as a problem that keeps handing itself a smaller version until the smallest one is easy to solve.

Underneath, both models do the same thing, predicting the next token, yet they behave very differently. Chat products make instruction following seem natural, which hides the gap. Questions get answered, code requests produce code, and summaries get written. None of that comes for free from reading a mountain of text. Pretraining instills something more basic: given everything so far, which token is probable next? Nothing in it says a request from a user should be carried out. Bridging that gap is the topic here.

Act 1: pretraining teaches completion, not obedience

The pretraining task is just next-token prediction. That alone is powerful enough to pick up grammar, facts, code syntax, writing styles and some surprisingly intricate patterns, but obeying instructions isn't among them. Feed a base model this prompt:

Explain why the sky appears blue.

and it might continue with:

Explain why the sky appears blue.
The following questions are commonly asked in introductory physics...

That isn't necessarily a broken model. It is doing precisely what pretraining asked, which is to continue the text. The desired behavior looks more like this:

The sky appears blue because Earth's atmosphere scatters
shorter wavelengths of sunlight more strongly than longer
wavelengths...

The underlying model is the same, but the expected behavior differs. The question changes from this:

What text probably comes next?

to this:

When the text looks like an instruction,
what kind of continuation should I produce?

Teaching the model to answer that second question is what instruction fine-tuning is for.

Act 2: formatting instruction data

The most direct way to teach instruction following is to show the model many instructions paired with good responses, and it really is that plain. A training record may look like this:

example = {
    "instruction": "Convert the following sentence to passive voice.",
    "input": "The developer fixed the bug.",
    "output": "The bug was fixed by the developer."
}

A transformer can't ingest Python dictionaries, so the record is first serialized into plain text, along these lines:

Below is an instruction that describes a task.

### Instruction:
Convert the following sentence to passive voice.

### Input:
The developer fixed the bug.

### Response:
The bug was fixed by the developer.

The formatting matters because the model repeatedly sees the same pattern:

instruction
    ↓
optional context
    ↓
response

In short, each record becomes formatted text, then tokens, then padded batches, and these batches fine-tune the pretrained GPT model.

The token view

Once tokenized, the example turns into a list of IDs, roughly like this:

[21106, 318, 281, 12064, ... 4435, 257, 3126, ...]

Those token IDs form the training sequence. For an autoregressive model, the labels are simply the sequence shifted by one position:

Tokens:
[A, B, C, D, E]

Input:
[A, B, C, D]

Labels:
[B, C, D, E]

So instruction tuning brings in no brand-new learning objective. The model still predicts the next token. What changes is the distribution of text it is asked to predict. Rather than mostly ordinary documents, it now sees this mapping again and again:

instruction → useful response

That is the central idea.

Padding needs special handling

Training examples vary in length, yet batches need tensors with compatible shapes. Take two examples:

Example 1:
[10, 20, 30, 40]

Example 2:
[11, 21]

They can be padded:

Example 1:
[10, 20, 30, 40]

Example 2:
[11, 21, PAD, PAD]

Padding tokens aren't meaningful targets. The model shouldn't be rewarded for learning this:

PAD → PAD → PAD → PAD

So those positions are usually replaced with an ignore value:

ignore_index = -100

PyTorch's cross-entropy loss then skips them. Conceptually:

loss = cross_entropy(
    logits.reshape(-1, vocab_size),
    targets.reshape(-1),
    ignore_index=-100,
)

The detail matters. Without masking, part of the training signal is wasted on artifacts created by batching rather than on language.

Act 3: fine-tune the behavior

The training loop itself isn't exotic here, since optimizing a next-token model is already familiar. What matters is which examples feed the loop. Simplified:

for batch in train_loader:
    optimizer.zero_grad()

    input_ids = batch[:, :-1]
    targets = batch[:, 1:]

    logits = model(input_ids)

    loss = cross_entropy(
        logits.flatten(0, 1),
        targets.flatten(),
        ignore_index=-100,
    )

    loss.backward()
    optimizer.step()

Nothing in that code says anything like this:

model.learn_to_be_helpful()

Behavior comes from the data. Suppose thousands of examples look like this:

### Instruction:
Summarize this paragraph.

### Response:
<clear concise summary>

gradient updates keep nudging the model toward replies of that shape after prompts of that shape. Suppose other examples look like this:

### Instruction:
Write Python code that reverses a string.

### Response:
def reverse_string(s):
    return s[::-1]

it learns another mapping between an instruction pattern and a fitting continuation. Viewed at token level, instruction fine-tuning is less magical: the context around the predictions has been deliberately changed. Before:

The CPU cache is...

After instruction formatting:

### Instruction:
Explain CPU caching to a beginner.

### Response:
A CPU cache is...

No goal such as "understand users" was introduced. Instead, the distribution of training text moved so that helpful answers turn into the statistically favored continuations after an instruction. That shift is what produces the new behavior. In the running example, a pretrained GPT-2-medium model gets instruction-fine-tuned, generates responses and saves the resulting model.

Act 4: evaluation gets uncomfortable

Classification offered an easy evaluation story. For spam detection:

Prediction: SPAM
Label:      SPAM

Correct.

Instruction following offers no such luxury. Consider the request to explain recursion simply. One possible answer:

Recursion is when a function solves a problem by calling
itself on a smaller version of the same problem.

Another:

Recursion is a technique where a problem is reduced into
smaller instances of itself until a stopping condition is reached.

Both can be correct, and neither has to match a reference word for word. That makes exact-match accuracy mostly useless for open-ended responses.

Comparing with a reference

One option is to supply a reference:

{
    "instruction": "Explain recursion simply.",
    "reference": "Recursion solves a problem by repeatedly reducing it...",
    "model_response": "A recursive function calls itself..."
}

and compare the generated response against it. But similarity isn't quality, and a model could give an excellent answer in completely different words. That leads to another approach.

Ask another LLM to judge

The evaluator receives the instruction, the reference answer and the response to rate:

Instruction:
Explain recursion simply.

Reference answer:
...

Model answer:
...

Rate the model answer from 0 to 100 based on correctness,
relevance, and clarity.

A local judge can run through Ollama, which is handy, yet it brings a fresh problem: the judge is itself a model. Its score can depend on the evaluation prompt, the choice of evaluator, response ordering, verbosity, stylistic taste and the judge's own mistakes. Some examples show a local judge handing out surprisingly generous scores to poor base-model responses, a reminder that LLM-as-a-judge numbers aren't ground truth.

A small project can stack three layers:

1. Inspect representative outputs manually
2. Compare against reference answers where useful
3. Use an LLM judge for aggregate comparison

None of these is perfect on its own. Together they give a clearer picture than a single number posing as truth.

Act 5: a harder problem, preferences

Instruction fine-tuning shows the model what a reasonable response looks like. Now compare two answers. Response A:

Recursion is when a function calls itself.

Response B:

Recursion is a technique where a function solves a problem
by calling itself on a smaller version of that problem.
The process stops when it reaches a base case.

Both are valid, but B may be preferred. The training signal is no longer this:

correct vs incorrect

It is this:

chosen vs rejected

A preference record can look like this:

{
    "prompt": "Explain recursion simply.",
    "chosen": """
    Recursion solves a problem by repeatedly reducing it
    until reaching a base case.
    """,
    "rejected": """
    Recursion is when recursion happens recursively.
    """
}

That is a different kind of supervision. Instead of saying "here is a response to imitate," it says "given these two responses, prefer this one." Here techniques such as Direct Preference Optimization (DPO) enter. At a high level:

Prompt
   ↓
Chosen response ─────┐
                     ├── preference objective
Rejected response ───┘
                     ↓
               model update

DPO deserves a separate treatment, so its mechanics stay out of this discussion. The useful link is the overall progression:

Pretraining
    ↓
learn language patterns

------------------------------

Instruction fine-tuning
    ↓
learn instruction → response behavior

------------------------------

Preference tuning
    ↓
learn which acceptable responses we prefer

Generating preference datasets with Llama 3.1 through Ollama and implementing DPO from scratch are topics for later. One caution applies: DPO being available as bonus material doesn't make it the right next step for every model or dataset. Some experiments show limits to its results, which is exactly the skepticism worth carrying into preference tuning.

What actually changed

The starting point was a pretrained model that knew how to continue text. The transformer wasn't swapped for a special assistant architecture. The examples around the next-token problem changed, producing a different behavioral prior. When the model now sees:

### Instruction:

it has met enough cases where the continuation isn't another paragraph from the internet but an answer. Seen this way, the progression is easier to reason about:

Pretraining:
What token should come next?

Instruction tuning:
What does a good answer look like after an instruction?

Preference tuning:
Of several reasonable answers, which behavior should we prefer?

Architecture, optimizer and learning rate all count, but in instruction tuning the data plays a special role, because it spells out the behavior to be imitated. With several acceptable answers on the table, valid output stops being the whole goal. The question turns into which acceptable response deserves more weight, and that is the door DPO opens.

Pretraining teaches what text tends to come next. Instruction tuning changes which continuations count as useful replies to instructions. Preference tuning takes the following step and decides which useful replies are the ones to prefer.