How an LLM Works: Tokens, Attention and Next-Token Prediction

Type a short sentence such as "The cat is sleeping." into a chat window, and a moment later the model answers with a line, an explanation or a whole paragraph. From outside it looks effortless. What actually happens in between?

Anyone who has read about AI has bumped into a pile of terms: tokens, embeddings, Transformers, attention, Q/K/V, next-token prediction. Together they can make language models sound far harder than they are. The good news is that the basic architecture doesn't need the underlying mathematics. It needs only an understanding of how a handful of pieces fit together.

One sentence will serve as the running example: "The cat is sleeping." Let's follow it through the machinery.

The big picture

Seen from far away, an LLM processes text in stages:

Text
 ↓
Tokens
 ↓
Embeddings
 ↓
Transformer
 ↓
Attention
 ↓
Next-token prediction
 ↓
Next token
 ↓
Repeat

Underneath all of it sits a large neural network whose parameters were set during training. Each stage is worth a closer look.

1. Neural networks, the foundation

A large language model is, at heart, one very big neural network. That's a machine-learning system that picks up patterns from data by tuning numeric parameters, usually called weights.

The simplest picture is a box that turns an input into an output:

Input
  ↓
Neural Network
  ↓
Prediction

While training, the network guesses, measures how far off it was, and nudges its parameters to guess better next time. After enough data, it has absorbed regularities of language. Given "The cat is ___", it learns that words like "sleeping", "hungry", "running" or "outside" are plausible continuations.

The point to keep: an LLM is a big neural network that learned patterns from data. You can skip the math of neural networks and still follow the rest.

2. Tokens, the way a model reads

You read words. The model doesn't handle words the way you do. The text is first chopped into smaller units called tokens. Conceptually it looks like this:

"The cat is sleeping."
          ↓
["The", " cat", " is", " sleeping", "."]

Which pieces you get depends on the model and its tokenizer. A token might be a complete word, a fragment of one, a punctuation mark or some other chunk of text. Every token is then mapped to a number, its token ID, so the pipeline so far goes:

Text
 ↓
Tokens
 ↓
Token IDs

At this point the model can handle the text as numbers.

Why tokens matter in practice

Tokens aren't a footnote. For anyone using AI products, they explain several everyday things:

• why models have a limit on context
• why long prompts consume more tokens
• why API usage is usually metered per token
• why padding a prompt with needless material raises the bill

For people building applications, tokens influence prompt design, context management and cost control.

Still, a bare ID isn't enough. The model wants something richer for each token.

3. Embeddings, meaning in numbers

Take the token "cat". Its ID might be something like 582, but that number says nothing about cats. It's a label and nothing more.

So the model swaps each ID for an embedding, a vector with many numeric values:

"cat"
  ↓
Token ID
  ↓
Embedding
  ↓
[0.21, -0.73, 0.44, ...]

An embedding is a numeric representation of a token that the neural network can compute with. The distinction is worth remembering: the token ID names a token, while the embedding describes it in numbers. These embeddings are what get fed into the Transformer.

4. Transformers, where the processing happens

By now the sentence has traveled this far:

Text
 ↓
Tokens
 ↓
Token IDs
 ↓
Embeddings

The embeddings enter the Transformer, a neural architecture built to handle sequences and, above all for language, to capture relationships between tokens. In simplified form:

Embeddings
    ↓
Transformer layer
    ↓
Transformer layer
    ↓
Transformer layer
    ↓
...
    ↓
Prediction

Real models stack many Transformer layers. But how does a layer work out which words matter to each other? Through self-attention.

5. Self-attention, using context

Look at a longer sentence: "The cat climbed onto the couch because it was tired." To handle the word "it", the model has to look around and decide what it points to. Self-attention lets it inspect other tokens and judge which are relevant:

The  cat  climbed  onto  the  couch  because  it  was  tired
      ↑                              ↑       ↑
      └──────── relevant context ────┘

You can phrase the mechanism as a question each token asks: which other tokens should I look at to understand myself? The model computes relationships between tokens and uses them to build representations that are aware of context. That ability is a major reason Transformers handle language so well.

6. Q, K and V, how attention picks what matters

Attention relies on three roles:

• Query (Q): what is this token searching for?
• Key (K): what does each token offer, as a label to match against?
• Value (V): what content does a token hand over when it's selected?

In flow form:

Query
  ↓
Compare with Keys
  ↓
Determine relevance
  ↓
Use relevant Values
  ↓
Updated representation

A library makes a good comparison. Your question is the query. Each book has a catalog entry that tells you whether it fits, and that's its key. Once you've found the relevant books, you read what's inside them, which are the values. Attention does something similar. The formula isn't needed to grasp the idea: Q, K and V help the model decide which information in other tokens is relevant.

7. Multi-head attention, many viewpoints

Language holds several kinds of relationships at once: grammar, meaning, and which words refer back to which. So Transformers run several attention heads side by side:

Input
                   ↓
          ┌────────┼────────┐
          ↓        ↓        ↓
        Head 1   Head 2   Head 3
          ↓        ↓        ↓
       Pattern   Pattern   Pattern
          └────────┼────────┘
                   ↓
                Combined

Each head can specialize in a different kind of relationship. One head offers one viewpoint, and several heads offer several. There's no need to understand each head's behavior individually to understand the overall design.

8. Causal masking, no peeking ahead

Now connect attention to text generation. Suppose the model sees "The cat is" and has to predict the next token. It must not be allowed to read the answer first. Causal masking enforces that:

The → can see The

cat → can see The, catis → can see The, cat, issleeping → can see The, cat, is, sleeping

Each position may use what came earlier but nothing that comes later. This matters most for GPT-style autoregressive models, for a plain reason: the model has to predict the future without being shown it.

9. Next-token prediction, how text gets produced

This is the central idea of GPT-style models. An LLM writes by predicting which token comes next. Given a start:

"The cat"
    ↓
Predict next token
    ↓
"is"

the context becomes:

"The cat is"

and it predicts again:

"The cat is"
       ↓
Predict next token
       ↓
"sleeping"

then once more:

"The cat is sleeping"
       ↓
Predict next token
       ↓
"."

and so on. A response generally isn't produced in one shot. The model loops:

Read the available context
        ↓
Predict the next token
        ↓
Add that token to the context
        ↓
Predict again
        ↓
Repeat

That's why next-token prediction is the idea to hold on to. Everything else in the architecture exists to make these predictions better.

10. Training and inference

One last distinction: learning versus using.

During training, the model sets its parameters. In outline:

Input
 ↓
Prediction
 ↓
Compare with expected result
 ↓
Calculate error
 ↓
Adjust parameters
 ↓
Repeat

This runs across vast amounts of data, and over time the model improves at predicting tokens and gathers useful patterns of language.

Inference is what happens whenever you use a trained model. You give it "The cat is", and it applies its learned parameters to predict the next token:

Your prompt
 ↓
Model
 ↓
Next-token prediction
 ↓
Next token
 ↓
Predict again
 ↓
Response

A short way to remember it: training is learning, and inference is using what was learned.

Everything together

Back to the opening sentence. Its path looks like this:

"The cat is sleeping."
          ↓
      Tokenization
          ↓
       Token IDs
          ↓
       Embeddings
          ↓
      Transformer
          ↓
    Self-Attention
          ↓
       Q, K, V
          ↓
  Multi-Head Attention
          ↓
     Causal Mask
          ↓
 Next-Token Prediction
          ↓
       Next Token
          ↓
         Repeat

A large neural network with learned parameters runs underneath the whole thing. That's the basic architecture.

Why it helps to know this

Using ChatGPT doesn't require knowing its internals. A rough mental model still tells you what these systems can and can't do. For ordinary users, it clarifies why context matters, why long prompts eat tokens, why a model may sound sure and still be wrong, and why supplying relevant information gets better results.

For builders, each concept pays off directly:

• Tokens explain context limits and costs.
• Embeddings lead to semantic search and RAG.
• Transformers and attention show how models handle context.
• Next-token prediction explains generation and behavior.
• The training-versus-inference split marks the line between using a model, fine-tuning one and training from scratch.

You don't have to master neural-network math before building with AI. Knowing the architecture does help you choose models, decide what information to supply, design the application and anticipate limits.

The one-paragraph version

If one thing sticks, let it be this: an LLM splits text into tokens, turns the tokens into numeric representations, runs them through Transformer layers that use attention, and keeps predicting the next token.

Text
 ↓
Tokens
 ↓
Embeddings
 ↓
Transformer
 ↓
Attention
 ↓
Next-token prediction
 ↓
Repeat

Once that flow is clear, topics like RAG, vector databases, fine-tuning, agents, evaluation and application development get easier, because you can see where each one slots into the picture.