GRPO Fine-Tuning Explained: Teaching a Local LLM to Reason

Reasoning-focused models such as DeepSeek-R1 have drawn a lot of attention, and many people now want the same behaviour in a small model they run themselves. The technique usually credited is GRPO fine-tuning. Supervised fine-tuning (SFT) teaches a model to imitate patterns in examples. Group Relative Policy Optimization (GRPO) uses reinforcement learning to push a model toward answers that actually work, which is closer to teaching it how to think than how to sound.

This piece walks through where GRPO sits among the RLHF methods, what its five core quantities mean, and how they turn into a few dozen lines of training code.

Where GRPO sits among preference-tuning methods

RLHF, reinforcement learning from human feedback, is best seen as an umbrella term. PPO, GRPO and DPO are different optimisers aimed at the same target: bringing a model in line with what people prefer.

A cooking comparison helps. RLHF is the idea of improving a dish by tasting and adjusting. PPO is an older, heavier recipe that needs a second model (a critic) and plenty of GPU memory. GRPO is a newer, leaner recipe that drops the critic and saves memory by comparing answers against each other.

The reinforcement-learning loop underneath

Generic RL runs in a loop: the agent sees a state, picks an action, the environment changes, a reward arrives and a new state follows. When PPO (Proximal Policy Optimization) was applied to language models, the language model became the agent, and the setup involved three pieces:

1. the generating policy, which is the model currently being trained
2. the reference policy, the original model kept frozen
3. the value model, which estimates an average reward

A reward model scores outputs, and training tries to raise that score.

PPO follows an actor-critic design, which means two separate networks are trained. The actor is the LLM itself. The critic is a value network of similar size that predicts the reward a response should earn. Having both roughly doubles the memory needed during training.

GRPO has no critic. By dropping it, memory use falls by something like half. For the baseline, GRPO generates a group of answers to the same prompt, scores each, and uses the group's average as the reference point for deciding which answers were better or worse than the rest.

DeepSeek introduced GRPO in its 2024 DeepSeekMath paper, and the method later fed into DeepSeek-R1.

The five quantities behind GRPO

Five ideas carry the method. For each, the aim is to explain what it measures and why it was designed that way.

1. The policy ratio: how far did the model's opinion move?

The ratio compares the probability the updated model gives to an answer with the probability the previous model gave to it: the new probability of response y for prompt x, divided by the old one.

Strip away the symbols and it asks how much more or less likely the model now is to produce this answer than before this training step.

Think of a student before and after a night of revision. Some answers now feel more obvious and others less so. The ratio captures that shift in confidence.

• Above 1 means the model is more confident in the answer than before.
• Below 1 means confidence has dropped and the model is drifting away from that answer.
• Exactly 1 means the update changed nothing for this answer.

Why does it matter? In plain reinforcement learning you might say a high-reward answer deserves a big probability boost. That's risky, because pushing too hard can make the model swing wildly and throw away good behaviour just because a single answer got lucky. The ratio works as a speed limit. GRPO clips large jumps so the model changes gradually and not erratically.

2. The mean reward: what did this batch score on average?

The mean is the sum of the rewards for the group of G answers, divided by G. G is the number of responses sampled for one question, and each response gets a score from the reward model.

Say you ask a model to solve a maths problem and sample five answers with some randomness. One is perfect (reward 1.0), another partly right (0.5) and a third plainly wrong (0.0). The mean tells you how well the model does on this problem right now.

The role it plays is what makes GRPO neat. PPO needed a critic, a second network whose whole job was to predict the expected reward, and training it was costly, memory-hungry and error-prone. GRPO asks why predict the average when you can compute it. Sampling a group and averaging the real rewards yields the baseline for free, with no extra network and no wasted memory. This is the "group" in the method's name.

3. The standard deviation: how spread out are the scores?

The standard deviation measures how far each reward sits from the mean, taking the square root of the average squared gap. If the mean describes typical performance, this describes consistency.

A high value means the responses are all over the place, some excellent and some awful. A low value means the model produces similar-quality answers every time.

Consider two students. Student A scores 90, 10, 95, 5 and 88, a mean of 57.6 with a huge spread: wildly inconsistent. Student B scores 58, 56, 60, 57 and 59, a mean of 58 with a tiny spread: dependably mediocre.

In GRPO the deviation works as a normaliser. When responses vary widely, differences between rewards carry real information and justify bigger updates. When the model is already consistent, small reward gaps may be noise, so updates should be more careful.

4. The group-relative advantage: did this answer beat the pack?

This is the centre of the method. The advantage for answer i is its reward minus the group mean, divided by the group's standard deviation. It converts raw rewards into a relative score that drives learning.

The subtraction says how much better or worse the answer is than average. The division scales by the spread so scores are comparable across questions and across training steps.

Picture grading a very hard exam where the class average is 40 out of 100. A student scores 60. As a raw mark that sounds mediocre, but relative to the class it's 20 points above average, which is excellent. The advantage works the same way, ignoring absolute reward values and asking whether this answer beat what the model usually produces for this prompt.

Three properties make this useful.

1. No critic. PPO's advantage depended on a critic estimating expected reward. GRPO swaps that guesswork for an empirical average over the group.
2. Self-calibration. If the reward model is biased and scores everything too high or too low, the relative form cancels that bias. Only the ranking inside the group matters.
3. Stability. Because of the normalisation, learning speed adapts. When the model is exploring widely, updates are bold. As it converges, they soften.

5. The KL penalty: stay recognisably yourself

This is the safety brake. While GRPO nudges the model toward high-reward answers, the KL term pulls the other way and says not to become someone else. It compares the current model with the frozen reference model, which is usually the base model from before RL began. KL divergence is a measure of how far apart two probability distributions are. The formula the source shows is a commonly used per-token estimate built from the ratio of reference to current probabilities.

Imagine the base model as a well-read generalist who can write poetry, debug code and summarise the news. Now you run RL to turn it into a maths whiz. Without a KL term, it may become so fixated on maths rewards that it forgets everything else, even producing maths-flavoured nonsense for unrelated prompts.

Low KL means the model still resembles its original self. High KL means it has wandered far, perhaps overfitting to the reward model or losing general ability.

Reward models are imperfect and can be gamed. A model with no KL term might discover that opening with "Let me think step by step" earns extra reward even when it adds nothing. Anchoring to the reference blocks that sort of reward hacking. In practice the KL term enters the loss with a small weight, around 0.01 to 0.04. It isn't there to stop progress, only to keep progress balanced.

The full update in plain words

1. Generate a group of answers to one question.
2. Score each with the reward model.
3. Compute the mean and standard deviation of those scores.
4. Work out each answer's advantage, meaning how far above or below average it landed.
5. Check the policy ratio to see how much confidence in each answer shifted.
6. Clip big changes so the model doesn't swing.
7. Penalise drift from the reference with the KL term.
8. Update the model to favour answers with high relative advantage.

The outcome is a model that learns to reason, not by memorising correct answers, but by discovering which patterns of thought regularly beat its own average.

From equations to trainer.train()

With the maths covered, here is code that puts it to work. It uses Unsloth to load a 4-bit Llama 3.2 3B Instruct model with a vLLM backend, attaches LoRA adapters, forces a reasoning format through a system prompt, and builds the training set from GSM8K. vLLM is there because GRPO needs to generate many completions per prompt quickly.

from unsloth import FastLanguageModel
from datasets import load_dataset
import re

# 1. Load 4-bit model with vLLM backend for fast group generation
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-3B-Instruct",
    max_seq_length=1024,
    load_in_4bit=True,
    fast_inference=True,
    max_lora_rank=32,
    gpu_memory_utilization=0.6,
)

# 2. Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    lora_alpha=64,
    use_gradient_checkpointing="unsloth",
)

# 3. Enforcing reasoning format - GRPO needs structure to reward
SYSTEM_PROMPT = "Respond in the following format:\n<reasoning>\n...\n</reasoning>\n<answer>\n...\n</answer>"

# 4. Load GSM8K and wrap prompts
dataset = load_dataset("openai/gsm8k", "main")["train"]
dataset = dataset.map(lambda x: {
    "prompt": [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": x["question"]}
    ],
    "answer": x["answer"].split("####")[1].strip()
})
#vLLM is used in GRPO because GRPO requires the AI to generate a ma

Reward functions

Reward design is the most delicate part of GRPO fine-tuning. Get it wrong and the model learns to game your scoring rules instead of reasoning. The individual scores are summed into the raw reward that feeds the advantage calculation. The set below has five functions, each with a reason to exist.

1. correctness_reward is the main signal. It extracts the text inside the answer tags and gives 2.0 for a match with the ground truth, otherwise 0.0. The weight is deliberately large so being right always outweighs looking tidy.
2. int_reward is a safety net. Early in training nearly everything is wrong, so it gives 0.5 just for producing a valid number, which keeps some learning signal alive until the model begins solving problems.
3. soft_format_reward and strict_format_reward reward the reasoning-then-answer tag structure with 0.5 each. The soft one tolerates layout differences, while the strict one demands exact placement. Together they coax the output from messy to clean.
4. xmlcount_reward pays 0.125 for each tag in the right place and subtracts a little for duplicate tags or trailing text after the answer. It's small polish that adds up.

Alone, each function is simple. Summed, correctness leads, format guides and the details look after themselves.

def extract_xml_answer(text: str) -> str:
    return text.split("<answer>")[-1].split("</answer>")[0].strip()

# Primary signal: is the final answer correct?
def correctness_reward(prompts, completions, answer, **kwargs):
    responses = [c[0]["content"] for c in completions]
    extracted = [extract_xml_answer(r) for r in responses]
    return [2.0 if r == a else 0.0 for r, a in zip(extracted, answer)]

# Dense shaping rewards - keep learning signal alive even when answers are wrong
def int_reward(completions, **kwargs):
    return [0.5 if extract_xml_answer(c[0]["content"]).isdigit() else 0.0 for c in completions]

def strict_format_reward(completions, **kwargs):
    pattern = r"^<reasoning>\n.*?\n</reasoning>\n<answer>\n.*?\n</answer>\nquot;
    return [0.5 if re.match(pattern, c[0]["content"]) else 0.0 for c in completions]

def soft_format_reward(completions, **kwargs):
    pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>"
    return [0.5 if re.match(pattern, c[0]["content"]) else 0.0 for c in completions]

def xmlcount_reward(completions, **kwargs):
    def score(text):
        count = 0.0
        if text.count("<reasoning>\n") == 1: count += 0.125
        if text.count("\n</reasoning>\n") == 1: count += 0.125
        if text.count("\n<answer>\n") == 1:
            count += 0.125
            count -= len(text.split("\n</answer>\n")[-1]) * 0.001
        if text.count("\n</answer>") == 1:
            count += 0.125
            count -= (len(text.split("\n</answer>")[-1]) - 1) * 0.001
        return count
    return [score(c[0]["content"]) for c in completions]pyt

The trainer

from trl import GRPOConfig, GRPOTrainer

trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    reward_funcs=[
        xmlcount_reward,      # tag quality
        soft_format_reward,   # structure (forgiving)
        strict_format_reward, # structure (exact)
        int_reward,           # partial credit
        correctness_reward,   # ground-truth match
    ],
    args=GRPOConfig(
        learning_rate=5e-6,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=1,
        num_generations=8,          # G = group size
        max_prompt_length=256,
        max_completion_length=200,
        max_steps=250,
        warmup_ratio=0.1,
        max_grad_norm=0.1,
        output_dir="outputs",
    ),
    train_dataset=dataset,
)

trainer.train()

At every step GRPOTrainer carries out the five calculations described above.

Why DPO struggles with reasoning

DPO is elegant because it sidesteps reinforcement learning altogether. It recasts the problem as classification: given a prompt and two answers, learn to prefer the winner. That works well when the preference labels are good.

Reasoning, though, isn't really a preference problem. It's a search problem.

Suppose you want a model to prove a geometry theorem. DPO needs a person or a stronger model to say which of two proofs is better. What if both are wrong? What if the correct proof takes 12 steps and was never sampled? DPO only learns from the pairs it's given and can't discover a new reasoning strategy.

GRPO generates a group of answers to the same prompt, scores each and asks which beat the average. That means it explores. It may hit on an unfamiliar line of reasoning, score it well and strengthen it, with nobody having written that route down beforehand.

Put simply, DPO is cheaper to run but limits the model to the quality of its preference data. GRPO costs more per step but can push beyond human-labelled data into reasoning that is genuinely new.

Closing thoughts

GRPO teaches a model to reason by learning which of its own responses beat its average. Removing the critic cuts memory use while keeping the learning signal intact, and with sensible reward functions plus group sampling you can improve a local model's reasoning on a single GPU. Reward the behaviour you actually want, keep the model anchored, and keep an eye on the KL value.