Temperature, Top-K, and Top-P: A Practical Guide to LLM Sampling

You push a chatbot live. People enjoy it. Then one afternoon someone posts a screenshot of a reply so bizarre it reads like the model was having a fever dream. At the same time, your coding assistant keeps returning the identical generic answer no matter how you phrase the question, like a parrot that memorized a syllabus.

Both symptoms trace back to one cause: the sampling settings were never adjusted.

Behind the scenes, the model is running through thousands of probability calculations, one token at a time.

• Every word.
• Every comma.
• Every next idea.

And here is what most developers overlook:

How creative, accurate, or predictable a response turns out can shift dramatically depending on just three settings: Temperature, Top-K, and Top-P.

These three knobs act as the steering wheel for how a language model behaves.

Whether you're working with OpenAI's API, Anthropic's models, Google Gemini, or Meta's Llama family, getting a real feel for these parameters changes how you build with them.

Let's go through each one, not in an academic way, but in a way you can actually apply.

First: How LLMs Actually Generate Text

Before diving into Temperature, Top-K, and Top-P, there's one core idea to grasp.

A language model doesn't compose sentences the way a person does. It predicts, one token at a time.

Take this prompt as an example:

"JavaScript is"

The model computes a set of candidate next tokens along with probabilities:

awesome -> 30%
a -> 25%
the -> 15%
used -> 10%
not -> 8%
weird -> 7%
broken -> 5%

These numbers come from patterns the model absorbed during training on enormous amounts of text.

The real question is: which of these tokens does it actually choose?

That decision is governed by sampling parameters, which act like filters on the raw probability list.

If a model simply always chose the single highest-probability token, that approach is called greedy decoding. It sounds sensible, but it performs poorly for creative writing, conversation, and any task that needs nuance, since it tends to produce dull, repetitive, overly cautious text.

Instead, models typically sample from the probability distribution, meaning they choose tokens based on likelihood rather than always grabbing the top-ranked one.

Temperature: The Creativity Dial

Temperature is the best-known of the three parameters, and also the one people misunderstand most often.

The core idea: temperature changes how sharp or how spread out the probability distribution becomes before a token is sampled.

• Low temperature (roughly 0.1 to 0.4): Sharpens the distribution. The already-likely token becomes even more dominant, so the model behaves more predictably, conservatively, and consistently.
• High temperature (roughly 0.8 to 1.5 and above): Smooths out the distribution. Tokens that were unlikely get a real shot at being picked, so output becomes more inventive, more surprising, and occasionally incoherent.
• Temperature = 0: Completely deterministic. The model always outputs the single most probable token — greedy decoding, essentially.
• Temperature = 1.0: No adjustment at all. The model samples using the original, unmodified probabilities.

The math (don't worry, it's simple)

Under the hood, temperature divides every raw model score (logit) before those scores get turned into probabilities:

adjusted_logit = original_logit / temperature

Those rescaled logits are then passed through a softmax function to produce the final probability distribution.

• Dividing by a small value (low temperature): Pushes logits further apart, sharpening the distribution, so the model commits to its top pick with more confidence.
• Dividing by a large value (high temperature): Squeezes logits closer together, flattening the distribution and injecting more randomness.

Example: Temperature = 0

Prompt: "Write a startup tagline for an AI coding tool."

Result: "Build software faster with AI."

Repeat the request ten times and you'll get the exact same line every time.

The reason is simple: temperature 0 always selects the single most probable token, with no exceptions.

This deterministic behavior is well suited to:

• Code generation
• SQL queries
• JSON output
• Structured data extraction

Example: Temperature = 0.3

Result: "Accelerate software development with intelligent AI."

Still fairly consistent, but with a touch more flexibility.

This range works well for:

• Technical writing
• Documentation
• API explanations

Example: Temperature = 1.0

Result: "Your AI co-pilot for turning midnight ideas into production code."

Now the output has more personality and variety. This range suits:

• Blog writing
• Marketing copy
• Brainstorming sessions

Example: Temperature = 2.0

Result: "Code dreams. Ship galaxies. Rewrite tomorrow with silicon imagination."

Is it imaginative? Sure.

Is it practical? Not really.

Push temperature too high and you risk drifting into text that doesn't make much sense.

When to Use What

Use                     | CaseTemperature
========================|=================
Code generation         | 0.0 – 0.2
Factual Q&A / RAG.      | 0.1 – 0.3
Summarization           | 0.3 – 0.5
Chatbot/conversation.   | 0.6 – 0.8
Creative writing        | 0.8 – 1.2
Brainstorming/ideation. | 1.0 – 1.5

What is Top-K?

Top-K restricts how many candidate tokens the model is even allowed to consider.

Rather than weighing the entire vocabulary, the model restricts itself to the K highest-probability tokens.

The rule is essentially:

"Discard everything outside the top K candidates."

With K = 50, the model only samples from the 50 most probable next tokens. Anything ranked 51st or beyond gets removed entirely, regardless of how likely it originally was.

Why This Exists

Picture a distribution spanning 50,000 possible tokens. Even tokens with tiny probabilities can occasionally get sampled, producing strange or nonsensical output. Top-K functions as a hard cutoff, essentially saying: "we won't even entertain the outliers."

Example

Prompt:

"React is"

Candidate next tokens:

Token    -> Probability
a        -> 35%
the      -> 20%
one      -> 15%
becoming -> 10%
fast     -> 8%
useful   -> 7%
wild     -> 5%

Top-K = 1

Only kept: [a]

Output: "React is a"

Extremely safe, with zero creative variation. This is functionally identical to greedy decoding.

Top-K = 3

Kept: [a, the, one]

This introduces some regulated variety.

Possible completions include:

• React is a…
• React is the…
• React is one…

A reasonable middle ground.

Top-K = 5

Kept: [a, the, one, becoming, fast]

More room to vary. Responses become noticeably more diverse.

Top-K = 50

A much wider net. Unusual but potentially interesting word choices can surface, though the chance of odd output also rises.

Real-world analogy for Top-K

Think about choosing food from a menu with 200 items.

Setting Top-K to 5 means you only glance at the five best-recommended dishes.

The decision becomes faster and less overwhelming. That's essentially the job Top-K performs for a language model.

What is Top-P? (Nucleus Sampling)

Top-P takes a more refined approach than Top-K.

Rather than truncating at a fixed number of candidates, it truncates based on accumulated probability. You keep adding tokens in order from most to least likely until their combined probability reaches P, then sample only from that set.

So if you set P = 0.9, the model draws from the smallest possible group of tokens whose probabilities together account for 90% of the total distribution.

Why "Nucleus Sampling"?

The name reflects the idea that the highest-ranked tokens form a core, a "nucleus", of realistic continuations. Everything outside that core is treated as noise: low-probability tail tokens the model technically scored but that shouldn't be genuine candidates for sampling.

Example probabilities:

Top-P = 0.50

Keep adding tokens until the running total hits at least 50%.

A = 40% B = 25%

Combined = 65%

Kept: [A, B]

Top-P = 0.80

A = 40% B = 25% C = 20%

Combined = 85%

Kept: [A, B, C]

Top-P = 0.95

A+B+C+D = 95%

Kept: [A, B, C, D]

The key thing to notice is that Top-P resizes its candidate pool on the fly.

That's precisely why it's often favored over Top-K in modern setups.

Typical Values

P    | ValueBehavior
=====|============================
0.5. | Very conservative, focused
0.75 | Balanced
0.9  | Standard creative tasks
0.95 | More exploratory
1.0  | No filtering (use all tokens)

Top-K vs Top-P

Top-K works with a fixed number of tokens.

Top-P works with a variable number of tokens.

Example:

When the probability distribution is sharply peaked, Top-P might end up keeping just 2 tokens.

When it's spread out, Top-P might keep 15.

That adaptability is what makes it so effective.

Top-K tells the model "choose from these X options." Top-P tells it "choose from however many good options exist."

That distinction matters a lot in practice.

Using Them Together (This Is Where It Gets Real)

Here's something rarely spelled out clearly: temperature, Top-K, and Top-P are applied in sequence, not in isolation.

A typical sampling pipeline looks like this:

Raw logits
    ↓
÷ Temperature  (reshape the distribution)
    ↓
Apply Top-K    (cut to top K tokens)
    ↓
Apply Top-P    (cut to nucleus)
    ↓
Sample         (pick one token from what's left)

Each stage narrows down the pool of candidate tokens further, and the order in which these filters are applied matters.

Practical Recipes for Real Projects

Recipe 1: The Code Assistant

temperature = 0.1
top_p = 0.95
top_k = 40

Here you want predictability, a single correct answer, and no surprises. A low temperature does most of the work, while Top-P acts as a backup guardrail.

Recipe 2: The Chatbot

temperature = 0.7
top_p = 0.9
top_k = 50

This produces conversational, natural-sounding replies without veering into randomness. The model reads as human, not robotic.

Recipe 3: The Creative Writing Partner

temperature = 1.1
top_p = 0.95
top_k = 0  # disabled

Give the model room to explore. You're after genuine creative variation, not generic filler. Top-K is switched off entirely, letting Top-P handle the trimming on its own.

Recipe 4: The Factual RAG System

temperature = 0.0
top_p = 1.0
top_k = 1

This is fully greedy decoding. Since the model already has the relevant context, you want the single most probable answer, no sampling randomness anywhere near something like an invoice pipeline.

Recipe 5: Blog Writing

Temperature = 0.8
Top-K = 40
Top-P = 0.95

This produces text that feels natural and engaging, suitable for long-form articles.

Recipe 6: Story Writing

Temperature = 1.2
Top-K = 100
Top-P = 0.98

This favors imaginative, less predictable output, well suited to fiction.

Recipe 7: Data Extraction

Temperature = 0
Top-K = 1
Top-P = 1

This is strict and fully deterministic, ideal for tasks like JSON extraction, classification, or entity extraction.

The Gotchas Nobody Warns You About

1. Higher temperature does not mean smarter output

It's tempting to raise temperature expecting the model to "think harder" or get more creative. What actually happens is that you inject noise. The model begins drawing from tokens it scored as unlikely, and usually for good reason. The result is hallucinated content, non-sequiturs, and broken grammar. Randomness-driven creativity is not the same thing as creativity that comes from sound reasoning.

2. Temperature = 0 is only deterministic within a single session

At temperature 0, the model effectively performs argmax, always picking the single highest-probability token. But across different hardware, batch sizes, or API versions, tiny floating-point rounding differences can still produce slightly different outputs. Don't assume perfect reproducibility unless you're also pinning a random seed where the API supports it.

3. Top-P and Top-K can work against each other

If Top-K is set very low, say K=5, while Top-P is set high, say P=0.95, Top-K effectively wins the argument. You've already restricted sampling to 5 tokens, so Top-P has no additional pool left to trim. Be deliberate about which parameter is actually doing the real filtering in your configuration.

4. Defaults differ across providers

OpenAI's GPT-4 ships with temperature=1.0 and top_p=1.0 by default. Anthropic's Claude models don't publish a single universal default, it varies by model version. Google's Gemini often uses temperature=1.0, top_p=0.95, and top_k=40 in some configurations.

Always verify these values yourself rather than assuming. Porting the same application to a different model without retuning these parameters can lead to noticeably different behavior.

Recommended Defaults

A reasonable general-purpose starting point looks like this:

Adjust these values based on the specific task at hand.

The Mental Model You Should Remember

If nothing else sticks, remember this:

Temperature governs randomness. Top-K governs how many options exist. Top-P governs the probability boundary of those options.

That's the whole picture.

Once you understand how these three interact, you stop merely "using" an AI model.

You begin engineering its outputs deliberately. That shift is what separates casual prompting from production-grade AI systems.

And going forward, that distinction only becomes more important.

Final Thoughts

Many developers spend all their effort refining prompts, but prompts are only one half of the equation. Sampling parameters are the less visible controls doing just as much work.

Often, they matter even more than the wording of the prompt itself.

The next time an LLM produces something strange, don't immediately blame the model.

Check instead:

• Temperature
• Top-K
• Top-P

Sometimes the model isn't the confused party.

Your configuration is. Once you internalize that, you gain a much deeper level of control over how these systems behave.

Happy Coding!