How Transformers Work, From RNN Bottlenecks to Attention, Agents and Context Windows
A step-by-step tour of the transformer: why RNNs stalled, how tokens and embeddings work, attention, QKV, heads, positions, block structure, scale and agents.
This article is published in English.
Anyone who only uses AI tools can get by without knowing what runs underneath. Anyone who wants to build, debug or push on those systems cannot. Every large language model in common use rests on one architecture, introduced in a 2017 paper of about fifteen pages titled "Attention Is All You Need". Few research papers of the past decade changed the direction of artificial intelligence as much. The ideas behind it are less forbidding than they sound, and the arithmetic is approachable when it is built up in the right order. After that, the same ideas turn up everywhere: in how agents plan, in why in-context learning works, in why a model can jump between topics mid-conversation, and in why longer context windows matter so much.
The route below starts from the old sequential approach and why it stalled, moves through how text becomes numbers, then through attention and the full block structure, and ends at how the design grew into today's language models and agents.
Why reading text one word at a time hit a wall
Before transformers, language was handled mostly by recurrent neural networks, usually shortened to RNNs. The idea is tidy. The network walks through a sequence word by word and carries a hidden state forward, a running summary of everything seen so far.
A rough picture: someone reads a novel but may keep only one sticky note, and has to go page by page, never skipping ahead or flipping back. After each sentence the note is rewritten with whatever seems important, but the note is small. By page 300 almost nothing from page 1 survives. Handing chapter two to a friend to read at the same time is also impossible, since the notes from chapter one have to be finished first.
That strict ordering causes two separate scaling problems.
The first is the vanishing gradient. Training adjusts weights by backpropagation, which traces how much each parameter contributed to the error. In an RNN reading a long sequence, that signal has to travel backward through every time step. By the time it reaches the earliest steps it has often shrunk to almost nothing, so the network effectively cannot learn from what those early words did.
The second is the sequential bottleneck. The result at position n cannot be computed before position n-1 is finished, because each step consumes the previous hidden state. Modern GPUs are built to do huge numbers of operations at once, and an RNN gives them almost nothing to do in parallel. A 1,000-word document takes 1,000 steps in a row, with the hardware mostly idle between them.
The diagram below contrasts the two ways of handling tokens, one at a time with a fading hidden state on one side and every pair connected at once on the other:
╔══════════════════════════════════════════════════════════════════════════════╗
║ RNN vs TRANSFORMER: HOW TOKENS ARE PROCESSED ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ ║
║ ┌─────────────────────────────────────────────────────────────────────┐ ║
║ │ RNN ── Sequential (slow, forgets early context) │ ║
║ └─────────────────────────────────────────────────────────────────────┘ ║
║ ║
║ "The" "cat" "sat" "on" "the" "mat" ║
║ │ │ │ │ │ │ ║
║ ▼ ▼ ▼ ▼ ▼ ▼ ║
║ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ║
║ │ RNN │──▶│ RNN │─▶│ RNN │▶│ RNN │▶│ RNN │▶│ RNN │ ║
║ │ 1 │ │ 2 │ │ 3 │ │ 4 │ │ 5 │ │ 6 │ ║
║ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ ║
║ h₁ ──▶ h₂ ──▶ h₃ ──▶ h₄ ──▶ h₅ ──▶ h₆ ║
║ ║
║ Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 (must wait!) ║
║ ⚠ "The" signal is diluted through 5 hops by the time we reach "mat" ║
║ ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ ║
║ ┌─────────────────────────────────────────────────────────────────────┐ ║
║ │ TRANSFORMER ── Parallel (fast, every token sees all others) │ ║
║ └─────────────────────────────────────────────────────────────────────┘ ║
║ ║
║ "The" "cat" "sat" "on" "the" "mat" ◀── ALL AT ONCE ║
║ │╲╲ │╲ ╲╲ │╲╲ ╲ │ ╲╲ │ ╲╲ │ ║
║ │ ╲╲ │ ╲ ╲╲ │ ╲╲ ╲ │ ╲╲ │ ╲╲ │ ║
║ ├───╫──┼───╫───┼───╫───┼────╫──┼────╫───┤ ← every token ║
║ │ ╱╱ │ ╱ ╱╱ │ ╱╱ ╱ │ ╱╱ │ ╱╱ │ attends to ║
║ │╱╱ │╱ ╱╱ │╱╱ ╱ │ ╱╱ │ ╱╱ │ every token ║
║ ▼ ▼ ▼ ▼ ▼ ▼ ║
║ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ║
║ │ out │ │ out │ │ out │ │ out │ │ out │ │ out │ ← all computed ║
║ │ 1 │ │ 2 │ │ 3 │ │ 4 │ │ 5 │ │ 6 │ simultaneously ║
║ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ ║
║ ✓ "The" has DIRECT connection to "mat" — no information decay ║
║ ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ LEGEND: ──▶ sequential dependency ╲╱╱ all-pairs attention ║
║ h₁ hidden state ⚠ information bottleneck ║
╚══════════════════════════════════════════════════════════════════════════════╝
Gates helped, but did not change the shape of the problem
Long Short-Term Memory networks, introduced by Hochreiter and Schmidhuber in 1997, added gates, explicit controls over what to keep, what to drop and what to emit at each step. They worked well. Through the 2010s they carried most practical NLP, including early machine translation, sentiment analysis and text generation.
They stayed sequential all the same. Very long-range dependencies were still hard, and parallel hardware still could not be used to its full extent. LSTMs did not fail because they modelled language badly. They failed because they could not scale: adding compute gave smaller and smaller returns, and a way past that limit needed a different style of computation altogether.
From text to numbers
Neural networks consume numbers and language is made of words, so a conversion has to happen before attention enters the picture. It takes two steps, tokenization and embedding.
Tokenization
Tokenization turns a string into a sequence of integer IDs drawn from a fixed vocabulary. The first thing to absorb is that tokens are not words. Modern tokenizers split text into subword pieces, usually with Byte-Pair Encoding or WordPiece, and the results look like this:
"Hello world" → ["Hello", " world"] → [15496, 995]
"unbelievable" → ["un", "believ", "able"] → [256, 9391, 540]
"ChatGPT" → ["Chat", "G", "PT"] → [14126, 38, 2969]
"antidisestablishmentarianism" → ["ant", "idis", "est", "ablish", "ment", "ari", "an", "ism"]
Frequent words get a token of their own, while rare or compound words are cut into smaller pieces. That gives good coverage with a vocabulary of manageable size, since there is no need for an entry for every possible word. Vocabulary sizes run from about 30,000 for BERT and 50,257 for GPT-2 to above 100,000 in recent models.
A handy conversion for English: a token is about three quarters of a word, or a word is about 1.3 tokens. A 100,000-word novel therefore comes to roughly 133,000 tokens, which matters whenever context-window limits are in play.
Embeddings
An embedding matrix with one row per vocabulary entry and d_model columns maps every token ID to a dense vector. Looking up a token is just selecting its row. Token ID 995, which stands for "world", points to row 995 and returns d_model numbers, for instance 0.24, -0.83, 0.17 and so on up to 0.91.
Those rows are learned during training, and a useful property emerges: tokens with related meanings end up with vectors that are geometrically close.
Cosine similarity in embedding space:
similarity("cat", "lion") ≈ 0.89 ← very similar
similarity("cat", "moon") ≈ 0.12 ← not similar
similarity("king", "queen") ≈ 0.91 ← very similar
One caution about the famous king minus man plus woman equals queen example. That arithmetic comes from Word2Vec in 2013 and not from transformer embeddings. Transformers learn their embeddings jointly with all the other parameters, so the clean vector arithmetic is less dependable. What does carry over is the principle that words with related meanings occupy neighbouring regions of the space.
One width everywhere
A fixed dimension, d_model, shows up everywhere in a transformer: embeddings, attention outputs, and the inputs and outputs of the feed-forward layers. That is deliberate. Every component reads from and writes to one fixed-width vector stream, called the residual stream, which flows through every layer. Because all components work on the same-sized vector and add their results back into it, the pieces compose cleanly. It is also the reason positional information can simply be added to a token embedding: both have the same shape, and the sum enters the stream directly.
So the input to the first block is the token embedding plus a position vector. Since both live in the same space, the addition costs no extra parameters, and position information is folded into the same vectors that carry meaning. BERT-style models that read two sentences together, as in question answering with a question and a passage, add a segment embedding that marks which sentence each token belongs to, 0 for the first and 1 for the second. GPT-style models skip it.
Attention: learning where to look
Around 2014 and 2015, researchers working on machine translation noticed a flaw even in LSTM systems. The model squeezed everything it knew about the input sentence into a single fixed-size vector before it began generating output, which was another bottleneck. The obvious question was whether the model could glance back at the whole input at every step of generation. The answer to that question became the attention mechanism, and it changed how sequences are modelled.
Two everyday situations make the idea intuitive.
At a noisy party with ten conversations going on, the brain does not give every sound equal weight when following a friend. It boosts the friend's voice and direction and turns the rest down, without switching the rest off entirely. The weighting is continuous, not an on/off switch.
Given a class photo and asked to find the teacher, a person does not scan each pixel equally. The search heads straight for features suggesting an adult, such as height compared with the others and facial proportions. What is being looked for steers where the eyes go.
Computational attention turns these intuitions into four steps: have something to look for, which is the query; have a set of candidates that might be relevant; give each candidate a continuous relevance score; and combine the information, weighted by those scores.
A misconception worth clearing up early: attention does not mean the model fixes on one thing and ignores everything else. The output is a weighted average to which every element contributes, some of them far more than others. It is graded consideration and not binary focus.
The original, bolted-on form
In its first form, attention was an add-on for RNN and LSTM models. At each decoding step, the model would:
- look at all of the encoder's hidden states;
- compute a relevance score between the current decoder state and each encoder state;
- turn the scores into weights with softmax;
- take the weighted sum of encoder states as the attention context.
Softmax deserves a short explanation. It converts a vector of arbitrary real numbers into a probability distribution. Each score becomes e raised to that score, divided by the sum of e raised to every score. All outputs are positive and they add up to 1, and because the exponential exaggerates the gap between large and small scores, the highest-scoring item gets the biggest share of the weight.
This was a big improvement, since the model could now reach any part of the input at each step. Then the authors of "Attention Is All You Need" asked something bolder: what if the recurrent machinery were removed and attention were not an extra, but the entire model?
Self-attention and the query, key, value trio
The central idea of the 2017 paper is self-attention. Rather than attending from one sequence to another, every token in a single sequence attends to every other token in that same sequence. That is how a transformer builds context, and understanding it requires three vectors found in every attention computation: the query, the key and the value.
A video search engine
Typing "how to make sourdough bread" into a video site does not compare the text against every frame of every video. Each video carries metadata labels, which play the role of keys: title, tags, description keywords. The site matches the query against those labels to get relevance scores, and the best-scoring videos are surfaced so their actual content, the values, can be retrieved.
In attention there is one important twist. The result is not a single video. It is a weighted blend of all videos at once, each played at a volume proportional to how relevant it is. The most relevant one dominates, with quieter contributions from the others layered underneath. The mapping is:
| YouTube Concept | Attention Concept | Question it Answers |
|----------------|------------------|---------------------|
| Search query | **Query (Q)** | "What am I looking for?" |
| Video metadata/tags | **Key (K)** | "What do I contain?" |
| Actual video content | **Value (V)** | "What should I contribute?" |
Why three separate vectors
Why not compare embeddings directly? Take the sentence "The cat sat on the mat because it was tired." To resolve what "it" refers to, the word has to attend to "cat". In raw embedding space, though, a pronoun and a noun are not especially similar. Splitting into queries and keys lets the model learn that a pronoun's query should match the keys of nouns it could refer to, even when the pronoun and the noun look nothing alike in their base embeddings. The learned projections build a relevance space that is separate from the similarity space of the raw embeddings. Relevance and similarity are different things, and the QKV design lets a model encode both.
Scaled dot-product attention in four steps
Step one is projection. Each token's embedding is multiplied by three learned matrices, W_Q, W_K and W_V, yielding a query, a key and a value for that token.
Step two measures compatibility. The question is how well token a matches token b, and the answer is the dot product of the query for a with the key for b. A high value means token a should pay close attention to token b. Dot products grow with vector size and with the key dimension d_k, not just with direction, which is why the next step rescales them.
Step three scales and normalises. When d_k is large, raw dot products become large and push softmax into regions where gradients are tiny. Dividing by the square root of d_k before softmax prevents that, which gives the method its name, scaled dot-product attention. The result is a set of weights alpha_ab.
Step four aggregates. The output for token a is the sum, over every token b, of alpha_ab times the value vector for b.
All tokens can be handled together as matrices, in one compact line: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V. The diagram traces it for the three-word phrase "the cat sat":
╔══════════════════════════════════════════════════════════════════════════════╗
║ SCALED DOT-PRODUCT ATTENTION ── "the cat sat" ║
╚══════════════════════════════════════════════════════════════════════════════╝
STEP 1: Input embeddings - ALL tokens are stacked into one matrix X
┌─────────────────────────────────────────┐
│ "the"→x₁ "cat"→x₂ "sat"→x₃ │ X ∈ ℝⁿˣᵈ
└──────────────────┬──────────────────────┘
│ the SAME X feeds all three projections
STEP 2: Three separate linear projections (each applied to every token)
┌─────────────┴─────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ W_Q │ │ W_K │ │ W_V │ ← learned weight matrices
└────┬────┘ └────┬────┘ └────┬────┘
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Q rows │ │ K rows │ │ V rows │
│ q₁ │ │ k₁ │ │ v₁ │
│ q₂ │ │ k₂ │ │ v₂ │
│ q₃ │ │ k₃ │ │ v₃ │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
STEP 3: score = QKᵀ, scale by 1/√dₖ, then softmax row-wise
│ │ │
└──────┬──────┘ │
▼ │
┌─────────────────────┐ │
│ softmax(QKᵀ / √dₖ) │ │
│ the cat sat │ │
│ the[.72 .20 .08] │ │ each row
│ cat[.12 .71 .17] │ │ sums to 1.0
│ sat[.09 .28 .63] │ │
└──────────┬──────────┘ │
│ │
STEP 4: multiply weights by Values V (weighted sum)
└──────────┬─────────┘
▼
┌─────────────────┐
│ output for "cat"│ 0.12·v₁ + 0.71·v₂ + 0.17·v₃
└────────┬────────┘ ← rich context-aware vector
│
OUTPUT: contextualized token representations
┌──────────┐ ┌──────────┐ ┌──────────┐
│"the" │ │"cat" │ │"sat" │
│knows it │ │knows it │ │knows │
│precedes │ │sat, and │ │who was │
│"cat" │ │where │ │sitting │
└──────────┘ └──────────┘ └──────────┘
A worked case: the word bank
Take "She went to the bank to deposit money". While the transformer encodes "bank", self-attention lets that word inspect every other word at once. Heavy weight lands on "deposit" and "money", which pulls financial meaning into the representation of "bank". Context resolves the ambiguity.
╔══════════════════════════════════════════════════════════════════════════════╗
║ SELF-ATTENTION RESOLVES AMBIGUITY: What does "bank" mean? ║
╚══════════════════════════════════════════════════════════════════════════════╝
Sentence: "I went to the bank to deposit money"
t₁ t₂ t₃ t₄ t₅ t₆ t₇ t₈
ATTENTION WEIGHTS for token "bank" (t₅) attending to all other tokens:
Token │ Weight │ Visualization
──────────┼────────┼───────────────────────────────────────────
"I" │ 0.03 │ ░
"went" │ 0.04 │ ░
"to" │ 0.02 │
"the" │ 0.03 │ ░
"bank" │ 0.08 │ ░░
"to" │ 0.05 │ ░
"deposit" │ 0.38 │ ████████████████████████░░░░░░░░ ← strong!
"money" │ 0.37 │ ████████████████████████░░░░░░░ ← strong!
FULL ATTENTION HEATMAP (rows = query token, columns = key token):
Higher weight → more ░/▒/▓/█
Q\K │ I went to the bank to deposit money
──────────┼──────────────────────────────────────────────
I │ ▓ ░ ░ ░ ░ ░ ░ ░
went │ ░ ▓ ░ ░ ░ ░ ░ ░
bank │ ░ ░ ░ ░ ▒ ░ █ █ ← "bank"→"deposit","money"
deposit │ ░ ░ ░ ░ █ ░ ▓ ▒ ← "deposit"→"bank"
money │ ░ ░ ░ ░ █ ░ ▒ ▓ ← "money"→"bank"
BEFORE attention: "bank" = ambiguous (river bank? financial bank?)
AFTER attention: "bank" ≈ 0.38·v(deposit) + 0.37·v(money) + ...
→ "bank" representation is now strongly financial ✓
The model learned this WITHOUT any explicit disambiguation rules.
The weights in that illustration are made up for clarity and do not come from a real model. Actual patterns tend to be messier, and research on how attention weights relate to what a model is "thinking" is still going. The practical payoff is separate from the picture: self-attention for all tokens is a matrix multiplication, a single GPU operation. What costs an RNN n sequential steps costs a transformer one parallel operation, and that is why transformers scale.
Multiple heads, multiple lenses
One head computes one kind of relevance at a time, so what happens when several kinds of relationship matter together? Look at "The musician who performed at the festival won the award." Understanding it means tracking several dependencies at once: "won" with "musician" as subject and verb, "musician" with "performed" through the relative clause, "performed" with "festival" for location, and "award" with "won" as verb and object.
A single head would have to cram all of those into one set of weights. Multi-head attention instead runs the attention function h times in parallel, each time with its own learned projections for queries, keys and values. The h outputs are concatenated and passed through one more learned matrix, W_O, which maps back to the model dimension. Each head works in a smaller subspace, with d_k equal to d_model divided by h, so the total cost across all heads is roughly that of a single full-width head.
Picture a manuscript sent to an editorial committee. A grammar editor checks syntax, a style editor follows narrative flow, a fact-checker looks for semantic consistency and a structural editor watches long-range organisation. They all read at the same time and independently, and their notes are merged into one review. Multi-head attention behaves the same way.
╔══════════════════════════════════════════════════════════════════════════════╗
║ MULTI-HEAD ATTENTION (3 heads shown) ║
╚══════════════════════════════════════════════════════════════════════════════╝
Input Embedding x ∈ ℝᵈ_model
┌────────────────────────────┐
│ [ x₁ x₂ x₃ ... xₙ ] │
└──────────┬─────────────────┘
│
┌────────────────────┼────────────────────┐
│ │ │
▼ ▼ ▼
╔══════════════╗ ╔══════════════╗ ╔══════════════╗
║ HEAD 1 ║ ║ HEAD 2 ║ ║ HEAD 3 ║
║ "Syntax" ║ ║ "Semantics" ║ ║ "Position" ║
║ ║ ║ ║ ║ ║
║ W_Q¹ W_K¹ ║ ║ W_Q² W_K² ║ ║ W_Q³ W_K³ ║
║ W_V¹ ║ ║ W_V² ║ ║ W_V³ ║
║ ║ ║ ║ ║ ║
║ ┌──────────┐ ║ ║ ┌──────────┐ ║ ║ ┌──────────┐ ║
║ │ Scaled │ ║ ║ │ Scaled │ ║ ║ │ Scaled │ ║
║ │ Dot- │ ║ ║ │ Dot- │ ║ ║ │ Dot- │ ║
║ │ Product │ ║ ║ │ Product │ ║ ║ │ Product │ ║
║ │Attention │ ║ ║ │Attention │ ║ ║ │Attention │ ║
║ └────┬─────┘ ║ ║ └────┬─────┘ ║ ║ └────┬─────┘ ║
║ │ ║ ║ │ ║ ║ │ ║
║ learns: ║ ║ learns: ║ ║ learns: ║
║ subject-verb ║ ║ word meaning ║ ║ nearby-word ║
║ agreement ║ ║ similarity ║ ║ relationships║
╚══════╤═══════╝ ╚══════╤═══════╝ ╚══════╤═══════╝
│ head₁ ∈ ℝⁿˣ⁽ᵈ/ʰ⁾ │ head₂ │ head₃
└───────────────────┼───────────────────┘
▼
┌──────────────────────────────┐
│ CONCATENATE all heads │
│ [head₁ ║ head₂ ║ head₃] │ ∈ ℝⁿˣᵈ_model
└──────────────┬───────────────┘
▼
┌──────────────────────────────┐
│ Final linear projection W_O │ blends info from all heads
└──────────────┬───────────────┘
▼
┌──────────────────────────────┐
│ Multi-Head Attention Output │ ∈ ℝⁿˣᵈ_model (same shape as input)
└──────────────────────────────┘
What heads end up doing
Analyses of trained models have found heads that specialise. Some track syntax, such as subject-verb agreement and modifiers attached to nouns. Some work over long distances, linking pronouns back to their antecedents. Some are local and attend to adjacent words and bigram patterns. Some register relative distance between tokens, and some pick out thematic roles, meaning who did what to whom. Nobody programs this. Training rewards accurate predictions, and the network discovers that splitting the labour among specialised heads works.
A limit on that story: these categories come mostly from interpretability work on smaller models. In large models many heads cannot be read cleanly, and experiments show such models often survive pruning of many heads with little loss. The architecture makes specialisation possible but does not promise tidy human-readable divisions.
Giving words an address
Self-attention relates every pair of tokens, but doing so treats the sequence as a bag with no order. The score between any token a and token b depends on content alone, not on which comes first or how far apart they sit. Language is strongly order-dependent. "Dog bites man" and "Man bites dog" contain the same words and mean opposite things, and "not happy" says something very different from "happy not". Since a transformer processes all positions in parallel, nothing in it encodes order by default, so position has to be injected explicitly. That is the job of positional encoding.
Think of adding metadata to every building in a city. Each needs an address that pins down its place, expresses relative position (building 402 sits near 404) and works without knowing the whole city plan beforehand. GPS coordinates do this neatly. Positional encoding is the linguistic version: a mathematical coordinate added to each word's representation.
Sine and cosine waves
The original transformer used sinusoidal functions, with one frequency per pair of embedding dimensions. For position pos and dimension index k, sines and cosines of pos divided by a power of 10000 scaled by k fill the vector. Slow, low-frequency waves mark coarse position and fast, high-frequency ones mark fine position, and together they give each position a unique fingerprint.
╔══════════════════════════════════════════════════════════════════════════╗
║ SINUSOIDAL POSITIONAL ENCODING ║
╚══════════════════════════════════════════════════════════════════════════╝
GRID: exact values for a tiny d_model=8 - positions 0-4, dims 0-7
(frequency denominators are 1, 10, 100, 1000 for dim-pairs 0,1,2,3)
dim→ d=0 d=1 d=2 d=3 d=4 d=5 d=6 d=7
sin cos sin cos sin cos sin cos
pos↓ ──────────────────────────────────────────────────────
pos=0 │ 0.000 1.000 0.000 1.000 0.000 1.000 0.000 1.000 │
pos=1 │ 0.841 0.540 0.100 0.995 0.010 1.000 0.001 1.000 │
pos=2 │ 0.909 -0.416 0.199 0.980 0.020 1.000 0.002 1.000 │
pos=3 │ 0.141 -0.990 0.296 0.955 0.030 1.000 0.003 1.000 │
pos=4 │ -0.757 -0.654 0.389 0.921 0.040 0.999 0.004 1.000 │
↑ fast-changing ↑ medium-changing ↑ very slow
(high frequency) (mid frequency) (low freq)
unique per position gradual shift global structure
WAVE PATTERNS (each row = one dimension across positions):
dim 0 (sin): █ ▇ ▄ ▁ ▂ ▅ ▇ █ ▇ ▄ ▁ ← fast oscillation
dim 2 (sin): █ █ ▇ ▇ ▆ ▅ ▄ ▃ ▂ ▁ ▁ ← medium oscillation
dim 4 (sin): █ █ █ █ █ █ ▇ ▇ ▇ ▇ ▆ ← slow oscillation
dim 6 (sin): █ █ █ █ █ █ █ █ █ █ █ ← very slow oscillation
HOW IT'S USED: Added to the token embedding elementwise
┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐
│ Token Embedding│ │ Positional Enc. │ │ Final Input x_t │
│ "cat" │ │ position = 2 │ │ │
│ [0.2, 0.8, │ │ [0.9,-0.4, │ │ [1.1, 0.4, │
│ 0.1, 0.5, │ + │ 0.2, 1.0, │ = │ 0.3, 1.5, │
│ 0.9, 0.3] │ │ 0.0, 1.0] │ │ 0.9, 1.3] │
│ (meaning) │ │ (where in seq) │ │ (meaning+position)│
└─────────────────┘ └─────────────────┘ └──────────────────┘
WHY SINUSOIDS?
✓ Each position gets a unique encoding across all dimensions
✓ Distance between nearby positions is consistent and learnable
✓ Can represent any position mathematically (unlike learned embeddings)
✓ Model can attend to relative positions via linear combinations
Lower dimensions oscillate quickly and capture fine-grained position, while higher dimensions change slowly and capture global order. The vector is simply added to the token embedding.
Fixed waves, learned tables and rotation
The sinusoidal scheme has a useful mathematical property: the encoding at position pos+k is a linear function of the encoding at pos. Relative offsets are therefore represented consistently, which makes positional relationships easier to learn.
Learned position embeddings, a separate table with one entry per position as BERT uses, are simpler but cannot extrapolate past positions seen in training, whereas sinusoids can in principle represent any position. Even so, sinusoids do not solve long-context generalisation fully. Models still struggle with sequences much longer than their training data, because the attention patterns themselves also have to generalise to unseen distances.
Models such as Llama use Rotary Position Embeddings, or RoPE. Before dot products are taken, queries and keys are rotated by an angle proportional to their positions. In the product of a query at one position and a key at another, the relative rotation encodes the offset directly. Making relative position an explicit input to the attention score, rather than something implied and learned through the projection matrices, helps with long contexts.
The complete architecture
Now the pieces can be assembled. The diagram shows the modern Pre-LayerNorm form of the whole structure:
╔═══════════════════════════════════════════════════════════════════════════════════════╗
║ THE COMPLETE TRANSFORMER ARCHITECTURE (Pre-LayerNorm, modern) ║
╠═══════════════════════════════════════════╦═══════════════════════════════════════════╣
║ ENCODER (×N layers) ║ DECODER (×N layers) ║
╠═══════════════════════════════════════════╬═══════════════════════════════════════════╣
║ ║ ┌───────────────────────────────────────┐ ║
║ ║ │ Linear → Softmax → P(next token) │ ║
║ ║ └───────────────────────────────────────┘ ║
║ ║ ▲ ║
║ ║ ┌───────────────────────────────────────┐ ║
║ ║ │ ⊕ Add (residual) │ ║
║ ║ └───────────────────────────────────────┘ ║
║ ║ ▲ ║
║ ║ ┌───────────────────────────────────────┐ ║
║ ║ │ FFN( LayerNorm(x) ) │ ║
║ ║ └───────────────────────────────────────┘ ║
║ ║ ▲ ║
║ Encoder output → K, V (to decoder) ║ ┌───────────────────────────────────────┐ ║
║ ▲ ║ │ ⊕ Add (residual) │ ║
║ ┌───────────────────────────────────────┐ ║ └───────────────────────────────────────┘ ║
║ │ ⊕ Add (residual) │ ║ ▲ ║
║ └───────────────────────────────────────┘ ║ ┌───────────────────────────────────────┐ ║
║ ▲ ║ │ Cross-Attention │ ║
║ ┌───────────────────────────────────────┐ ║ │ Q ← decoder │ ║
║ │ FFN( LayerNorm(x) ) │ ║ │ K, V ← encoder output │ ║
║ └───────────────────────────────────────┘ ║ └───────────────────────────────────────┘ ║
║ ▲ ║ ▲ ║
║ ┌───────────────────────────────────────┐ ║ ┌───────────────────────────────────────┐ ║
║ │ ⊕ Add (residual) │ ║ │ ⊕ Add (residual) │ ║
║ └───────────────────────────────────────┘ ║ └───────────────────────────────────────┘ ║
║ ▲ ║ ▲ ║
║ ┌───────────────────────────────────────┐ ║ ┌───────────────────────────────────────┐ ║
║ │ Multi-Head Self-Attention │ ║ │ Masked Multi-Head Self-Attention │ ║
║ │ ( on LayerNorm(x), bidirectional ) │ ║ │ ( on LayerNorm(x), causal ) │ ║
║ └───────────────────────────────────────┘ ║ └───────────────────────────────────────┘ ║
║ ▲ ║ ▲ ║
║ ┌───────────────────────────────────────┐ ║ ┌───────────────────────────────────────┐ ║
║ │ Input Embedding + Positional Enc. │ ║ │ Output Embedding + Positional Enc. │ ║
║ └───────────────────────────────────────┘ ║ └───────────────────────────────────────┘ ║
║ ▲ ║ ▲ ║
║ ┌───────────────────────────────────────┐ ║ ┌───────────────────────────────────────┐ ║
║ │ Source: "The cat sat on the mat" │ ║ │ Target (shifted): <BOS> "Le" "chat"… │ ║
║ └───────────────────────────────────────┘ ║ └───────────────────────────────────────┘ ║
╠═══════════════════════════════════════════╩═══════════════════════════════════════════╣
║ Data flows UPWARD (▲). Each sub-layer is a residual branch over a ║
║ LayerNorm-ed input: h = x + SubLayer( LayerNorm(x) ) ║
║ The encoder's output (K, V) flows into the decoder's Cross-Attention. ║
╚═══════════════════════════════════════════════════════════════════════════════════════╝
A transformer has two main parts, an encoder and a decoder. In the original paper each had 6 layers, with 8 attention heads and d_model of 512.
The encoder: understanding in context
The encoder reads the input sequence and produces a contextualised representation of every token, capturing not what a word means in isolation but what it means in this particular sentence. It uses bidirectional self-attention, so each token can look at every other token on both sides. This is how the "bank" ambiguity above gets resolved: the model sees "deposit" and "money" at the same time.
Six encoder layers run in sequence, each refining the representation. Early layers lean toward syntax such as word order and part of speech. Middle layers handle more intricate grammatical structure. Later layers capture high-level meaning such as entity types, event structure and pragmatics. This layering is a reason transfer works so well: fine-tuning mainly reshapes the later layers, while the general linguistic patterns in early layers stay useful across tasks.
The decoder: generating with memory
The decoder emits output tokens one at a time, autoregressively. Each new token depends on everything generated so far plus the whole encoded input. It has three sub-layers.
The first is masked self-attention, also called causal attention. The decoder may look only at tokens it has already produced, enforced by a causal mask. Scores above the diagonal are set to minus infinity before softmax, which makes future tokens invisible. Training then matches inference, because the model has to predict every token from past context only.
╔══════════════════════════════════════════════════════════════════════════╗
║ DECODER SELF-ATTENTION: Causal (Autoregressive) Masking ║
║ "Each token can only attend to itself and tokens before it" ║
╚══════════════════════════════════════════════════════════════════════════╝
Tokens: [The] [cat] [sat] [on] [mat]
t₁ t₂ t₃ t₄ t₅
ATTENTION SCORE MATRIX (row = query, col = key)
KEY → t₁ t₂ t₃ t₄ t₅
QUERY ↓ "The" "cat" "sat" "on" "mat"
──────────────────────────────────────
t₁ "The" │ ✓ ✗ ✗ ✗ ✗
t₂ "cat" │ ✓ ✓ ✗ ✗ ✗
t₃ "sat" │ ✓ ✓ ✓ ✗ ✗
t₄ "on" │ ✓ ✓ ✓ ✓ ✗
t₅ "mat" │ ✓ ✓ ✓ ✓ ✓
✓ = ALLOWED (score kept, participates in softmax)
✗ = MASKED (score → −∞, becomes 0 after softmax)
When generating t₃ "sat":
Raw scores: [ 2.1 3.4 1.8 -∞ -∞ ]
After softmax: [ 0.18 0.68 0.14 0 0 ]
↑
future tokens are invisible
WHY IS MASKING NECESSARY?
┌──────────────────────────────────────────────────────────┐
│ TRAINING: We feed the whole target sequence at once │
│ (teacher-forcing). Without masking, t₃ could "cheat" │
│ by looking at t₄ and t₅ - the answers it should │
│ be predicting. │
│ │
│ INFERENCE: We generate one token at a time anyway, │
│ so future tokens literally don't exist yet. Masking │
│ makes training match inference behavior exactly. │
└──────────────────────────────────────────────────────────┘
The second is cross-attention, the bridge between encoder and decoder. Queries come from the decoder, representing the current state of generation, while keys and values come from the encoder, representing what the input said. Translation works this way. When the decoder is producing the French word "chat", it queries the encoder's representation of the full English sentence, and cross-attention directs heavy weight to "cat".
The third is the feed-forward network, identical in role to the one in the encoder and covered below.
Three families built from the same parts
The encoder-decoder pairing is only one way to use these components, and different tasks call for different arrangements:
| Model | Architecture | Training Objective | Best For |
|-------|-------------|-------------------|----------|
| **BERT** | Encoder-only | Masked Language Modeling | Understanding: classification, NER, QA |
| **GPT** | Decoder-only | Next token prediction | Generation: writing, coding, conversation |
| **T5** | Encoder-Decoder | Text-to-text seq-to-seq | Transformation: translation, summarization |
╔══════════════════════════════════════════════════════════════════════════════════════╗
║ BERT vs GPT vs T5: Architecture Families ║
╠══════════════════╦═══════════════════════════╦═════════════════════════════════════╣
║ BERT ║ GPT ║ T5 ║
║ Encoder-Only ║ Decoder-Only ║ Encoder + Decoder ║
╠══════════════════╬═══════════════════════════╬═════════════════════════════════════╣
║ ATTENTION: ║ ATTENTION: ║ ATTENTION: ║
║ t₁ ● ● ● ● ║ t₁ ● ║ Encoder: ● ● ● ● (bidir) ║
║ t₂ ● ● ● ● ║ t₂ ● ● ║ ● ● ● ● ║
║ t₃ ● ● ● ● ║ t₃ ● ● ● ║ Decoder: causal + ║
║ t₄ ● ● ● ● ║ t₄ ● ● ● ● ║ cross-attn to encoder ║
║ (bidirectional) ║ (causal/left-only) ║ ║
╠══════════════════╬═══════════════════════════╬═════════════════════════════════════╣
║ BEST FOR: ║ BEST FOR: ║ BEST FOR: ║
║ ✓ Classification ║ ✓ Text generation ║ ✓ Translation ║
║ ✓ NER / tagging ║ ✓ Completion ║ ✓ Summarization ║
║ ✓ Sentence embeds ║ ✓ Chat / dialogue ║ ✓ Question answering ║
║ ✓ Reading comp. ║ ✓ Code generation ║ ✓ Any seq-to-seq task ║
╠══════════════════╬═══════════════════════════╬═════════════════════════════════════╣
║ EXAMPLES: ║ EXAMPLES: ║ EXAMPLES: ║
║ BERT, RoBERTa ║ GPT-4o, Claude 4, ║ T5, BART, ║
║ DeBERTa ║ Llama 4, Gemini 2.5 ║ mT5, FLAN-T5 ║
╚══════════════════╩═══════════════════════════╩═════════════════════════════════════╝
BERT reads in both directions and suits understanding tasks. GPT generates left to right under causal masking. T5 keeps the full architecture for structured sequence-to-sequence transformation. The choice among them is about fitting architecture to task and not about which is better. In short, the encoder builds bidirectional contextual representations, the decoder generates autoregressively with causal masking and cross-attention, and different subsets of the structure are tuned for understanding, generation and transformation.
The supporting cast
Attention and the feed-forward layer get most of the attention, but three other parts keep deep transformers from collapsing in training: the feed-forward networks themselves, layer normalisation and residual connections.
Feed-forward networks: thinking time per token
After every attention layer in each block comes a position-wise feed-forward network. Each token's vector goes through the same two-layer MLP, applied separately to each position: multiply by W1, add a bias, apply a GeLU nonlinearity, then multiply by W2 and add another bias. W1 maps from d_model to four times d_model, and W2 maps back. In the original sizes that is 512 expanding to 2048 and contracting to 512, which gives the network room to store and transform information that attention alone does not capture.
"Position-wise" is the key phrase, since the layer treats each token independently. Attention is the communication mechanism, where tokens talk to one another, and the feed-forward network is the computation mechanism, where each token processes what it has gathered.
Geva and colleagues (2021) found that these layers behave a bit like key-value memories, and individual neurons can be identified that fire for particular semantic concepts. When a language model "knows" that Paris is the capital of France, that association lives mostly in feed-forward weights. This explains two things. Fine-tuning can update factual knowledge because it updates those weights. And retrieval-augmented generation works because it places the fact in the context window, where attention can reach it directly.
Residual connections
Gradients travelling backward through a deep network shrink over many layers. A residual connection, also called a skip connection, solves this neatly: the output of a sublayer is the sublayer's result plus its own input. The layer then only has to learn the change, the residual, and not the whole mapping. This builds a highway along which gradients travel straight back, bypassing any layer, and it is why modern transformers can stack 96 or more layers without training falling apart.
Layer normalisation: keeping scales in check
Layer normalisation standardises each token's activation vector to zero mean and unit variance, then applies a learned scale gamma and shift beta. Statistics come from the d_model features of one token: the mean is the plain average of its entries, and the variance is the average squared distance from that mean. A small constant epsilon, typically 10 to the minus 5, goes inside the square root so there is never a division by zero. Unlike batch normalisation, the computation is per token and not across the batch, which suits sequences of variable length.
The modern block pattern
The original paper applied LayerNorm after the residual addition, known as Post-LN. Nearly every modern model, from GPT-2 and GPT-3 to Llama and essentially everything trained since around 2020, uses Pre-LayerNorm and normalises before each sublayer. Pre-LN trains more stably and does not need learning-rate warmup.
Each block then adds an attention branch and a feed-forward branch around normalised inputs. The first intermediate state is the input plus attention applied to the normalised input, and the second is that state plus the feed-forward network applied to its own normalised version. Opening any modern transformer's code shows LayerNorm ahead of attention and ahead of the feed-forward step. Repeat that block N times, with N at 6, 12, 24, 96 or more depending on model size, and the whole transformer is there.
The attention mechanism is the clever idea. These supporting parts are the engineering that makes the clever idea work. Feed-forward layers provide per-token computation that stores factual associations, residual paths give gradients a route through depth, and Pre-LN keeps activations stable. Together they make stacking dozens of blocks feasible.
The path of a sentence
One run end to end goes: tokenize, embed, add position encoding, encode, decode one token at a time, then project to the vocabulary. The next diagram follows "Hello world" through a translation:
╔══════════════════════════════════════════════════════════════════════════╗
║ END-TO-END DATA FLOW: "Hello world" → Translation ║
╚══════════════════════════════════════════════════════════════════════════╝
STAGE 1: Tokenization
┌─────────────────────────────────────────────────────────────────┐
│ Raw text: "Hello world" │
│ ↓ Tokenizer (BPE / WordPiece) │
│ Token IDs: [ 7592, 2088 ] ← integers into vocabulary │
└──────────────────────┬──────────────────────────────────────────┘
↓
STAGE 2: Embedding Lookup e_t = E[t] ∈ ℝ^512
┌─────────────────────────────────────────────────────────────────┐
│ 7592 → [0.21, -0.54, 0.88, ...] "Hello" │
│ 2088 → [0.67, 0.12,-0.34, ...] "world" │
└──────────────────────┬──────────────────────────────────────────┘
↓
STAGE 3: Add Positional Encoding x_t = e_t + PE(t)
┌─────────────────────────────────────────────────────────────────┐
│ embedding + PE = input vector carrying BOTH meaning AND order │
└──────────────────────┬──────────────────────────────────────────┘
↓
STAGE 4: Encoder Stack (× N layers)
┌─────────────────────────────────────────────────────────────────┐
│ ╔═══════════════════════════════════╗ │
│ ║ Each Encoder Layer: ║ × 6 │
│ ║ h₁ = x + Attn(LayerNorm(x)) ║ │
│ ║ h₂ = h₁ + FFN(LayerNorm(h₁)) ║ │
│ ╚═══════════════════════════════════╝ │
│ After 6 layers: rich contextual reps - "Hello" knows context │
└──────────────────────┬──────────────────────────────────────────┘
↓ (K, V flow into decoder cross-attention)
STAGE 5: Decoder Stack (generates tokens autoregressively)
┌─────────────────────────────────────────────────────────────────┐
│ Step 1: <BOS> → Decoder → predict "Hola" │
│ Step 2: <BOS> "Hola" → Decoder → predict "mundo" │
│ Step 3: <BOS> "Hola mundo" → Decoder → predict <EOS> │
│ │
│ Each step: ① Masked self-attention on generated tokens │
│ ② Cross-attention: Q←decoder, K,V←encoder │
│ ③ LayerNorm → FFN → residual │
└──────────────────────┬──────────────────────────────────────────┘
↓
STAGE 6: Output Projection & Softmax
┌─────────────────────────────────────────────────────────────────┐
│ hidden state → W_vocab → logits over 50k vocab items │
│ → softmax → probability distribution │
│ → sample or argmax → next token │
└──────────────────────┬──────────────────────────────────────────┘
↓
"Hola mundo" ✓
How a translation paper became a language model
That covers the complete architecture, but how did a design for machine translation turn into systems that write essays and code and work through hard problems? The short answer is scale.
The timeline below runs from the 2017 transformer to recent frontier models:
YEAR MODEL PARAMS KEY DEVELOPMENT
────────────────────────────────────────────────────────────────────────────────
2017 Transformer 65M "Attention Is All You Need" published
2018 GPT-1 117M First successful generative pre-training
2019 GPT-2 1.5B Improved fluency; staged release
2020 GPT-3 175B Few-shot in-context learning emerges
2022 ChatGPT 175B+ InstructGPT + RLHF = aligned conversational AI
2023 GPT-4 undisclosed Multimodal input + stronger reasoning
2024 GPT-4o undisclosed Natively multimodal, faster & cheaper (May)
2024 o1 undisclosed "Reasoning" model: thinks before answering (Sep)
2025 o3 / o4-mini undisclosed Stronger reasoning, tool use, long-horizon
2025 Claude 4 undisclosed Anthropic: Opus 4 / Sonnet 4 (May)
2025 Gemini 2.5 undisclosed Google: 1M-token context, multimodal
2025 Llama 4 undisclosed Meta: open weights, up to 10M-token context
2026 GPT-5 series undisclosed OpenAI flagship (GPT-5.4 / 5.5); 272K-1M ctx
2026 Gemini 3.x undisclosed Google: frontier reasoning; 1M context
2026 Claude 5 family undisclosed Anthropic: Opus 4.8, Sonnet 5, Fable 5
2026 Grok 4.x undisclosed xAI frontier line (Grok 4.5, Jul 2026)
(Every one is the same core transformer from 2017 — scaled and refined, not replaced)
The core architecture barely changed between GPT-1 and today's frontier models. What changed was the following:
- Scale, meaning more parameters, more data and more compute.
- Alignment, with reinforcement learning from human feedback making models helpful and safe.
- Instruction tuning, which trains models to follow directions and not just to continue text.
- Reasoning training, which applies large-scale reinforcement learning to verifiable problems. It is the ingredient behind the reasoning models, such as OpenAI's o-series and the extended-thinking modes now common in Claude, Gemini and others, which spend extra compute thinking before they answer.
Abilities that seem to switch on
Something unexpected happened as models grew. Certain abilities did not improve gradually and seemed to jump at particular sizes. Multi-step arithmetic was nearly absent below about 10 billion parameters and then reliable near 100 billion. Chain-of-thought reasoning was essentially missing unless the model was large enough and prompted for it. Code generation got dramatically better above GPT-3 scale. None of this was programmed, and it emerged.
How sharp those jumps truly are is debated. A widely cited 2023 analysis by Schaeffer and colleagues, "Are Emergent Abilities of Large Language Models a Mirage?", showed that some apparent step changes come partly from the metric. Scoring a task all or nothing, as exact-match accuracy does, can turn steady, gradual gains into what looks like a sudden leap, whereas a continuous metric shows the steady progress underneath. The practical point holds, since large models do things small ones cannot, but specific threshold numbers should be taken as rough illustration and not as hard physics.
The training objective of every GPT-style model is the same simple task of predicting the next token. Reasoning, code generation, multilingual understanding and creative writing are never trained explicitly. They emerge as the model gets better at next-token prediction, provided there is enough scale: a large, high-quality, diverse corpus and a model big enough to hold the necessary patterns. That is why researchers call the objective deceptively simple. The objective is simple, and what a model must learn in order to do well at it is not.
The context window as working memory
Working memory, for a model, is the context window: the largest number of tokens it can take in during a single forward pass. Growth has been dramatic. GPT-2 in 2019 handled 1,024 tokens and GPT-3 in 2020 handled 2,048. GPT-4o in 2024 reached 128,000. By mid-2026 the frontier had moved much further: OpenAI's GPT-5 series offers 272,000 tokens as standard and about a million through the API, Anthropic's Claude and Google's Gemini 3 offer over a million, and Meta's open-weight Llama 4 Scout advertises up to 10 million tokens, roughly 15,000 pages. Those figures keep rising with each generation.
A larger window matters because it allows analysing a whole code file and not only snippets, keeping a conversation coherent over many turns, summarising entire documents, and doing multi-step reasoning with many intermediate results in view.
There is a catch. Computing the full attention matrix, which relates every token to every other token, takes O(n squared) time and memory at training. Doubling the context length quadruples both. At inference, KV caching lowers the per-step compute, but memory still grows linearly with context length. This quadratic training bottleneck is an active research area. Sparse attention, sliding-window attention as used in Mistral and other efficient designs, Flash Attention (memory-efficient exact attention) and linear approximations all try to cut the cost.
What sits on top of the foundation
Attention, QKV, multiple heads, positional encoding and the encoder-decoder layout are infrastructure, and the reason to learn them is everything built over them.
Generative AI is next-token prediction repeated
Every generative application is, underneath, the same loop run again and again:
Prompt: "Write a poem about autumn"
↓
Model predicts most likely next token: "The"
↓
Prompt + "The" → predicts: "leaves"
↓
Prompt + "The leaves" → predicts: "fall"
↓
... continues until <end-of-sequence> token predicted
Seeing it that way takes the mystery out of several behaviours:
- Hallucination. The model predicts likely tokens and does not retrieve facts from a database. If a plausible but wrong token is available, it may be produced.
- Why context matters so much. The whole context window is attended to at every generation step, so whatever goes into the prompt shapes which attention weights are computed and thereby what comes out.
- What temperature does. Temperature is applied to the output logits, the scores across the whole vocabulary, before sampling. A higher value flattens the distribution so more tokens become plausible and output varies more, and a lower value concentrates probability on the likeliest token and makes output more deterministic. This is a separate thing from the fixed 1/sqrt(d_k) scaling inside attention.
In-context learning: attention doing retrieval
One of the most striking abilities of large transformers is in-context learning, adapting to a new task from examples in the prompt without any fine-tuning. The mechanism fits what was covered above. The few-shot examples occupy positions in the window. When the model processes the test instance, that token's query is compared by dot product with the keys of every earlier token, example pairs included. Because the examples share a structural pattern with the test instance, their keys match well and their values receive high weights. Attention is literally the retrieval mechanism, and that is why the structure of a prompt, through its effect on attention patterns, changes the output.
Agents: a transformer as the reasoning engine
AI agents are systems that decompose tasks on their own, call tools, iterate on feedback and pursue goals across many steps. In 2023 they were a research curiosity. Now products such as Claude Code, GitHub Copilot, Cursor and Devin are used daily by developers. Every serious agent has a transformer at its core.
╔══════════════════════════════════════════════════════════════════════════════╗
║ THE CONTEXT WINDOW AS AGENT WORKING MEMORY ║
╚══════════════════════════════════════════════════════════════════════════════╝
TIME ──────────────────────────────────────────────────────────────────────▶
┌──────────────┬──────────────┬────────────────┬────────────┬──────────────┐
│ SYSTEM │ USER │ AGENT │ TOOL │ AGENT │
│ PROMPT │ MESSAGE │ THOUGHT │ RESULT │ RESPONSE │
│ │ │ (reasoning) │ │ │
│ "You are a │ "What is │ <think> │ {"temp": │ "The current │
│ weather │ the weather │ I need to │ 22, │ temperature │
│ assistant │ in Paris │ call the │ "unit": │ in Paris │
│ ..." │ today?" │ weather API │ "C", │ is 22°C and │
│ │ │ for Paris... │ "cond": │ cloudy." │
│ │ │ </think> │ "cloudy"} │ │
│ ~200 tokens │ ~15 tokens │ ~80 tokens │ ~40 tokens │ ~25 tokens │
└──────────────┴──────────────┴────────────────┴────────────┴──────────────┘
▲ ▲ ▲ ▲ ▲
└───────────────┴───────────────┴───────────────┴─────────────┘
│
At each generation step:
Full self-attention runs over the
ENTIRE context simultaneously.
Cost: O(n²) in total context length.
WHAT THIS MEANS FOR AGENTS:
✓ Tool results are just tokens - the model attends to them like any text
✓ Reasoning traces are visible - chain-of-thought shapes next generation
✓ Long task history - "memory" without any separate mechanism
⚠ Context fills up - very long loops need summarization/offload
⚠ O(n²) attention cost - longer context = quadratically more compute
KEY INSIGHT: There is no "working memory" separate from the context.
The context window IS the agent's entire cognitive state.
For an agent, the window is one flat, ordered buffer of tokens, and every generation step attends across all of it, which keeps everything that has happened within reach up to the limit. The window is the agent's working memory, attention is its reasoning mechanism and the next-token loop is its action loop. Agentic AI needed no new architecture. What it took was noticing that the transformer, at large enough scale, could already support agency.
In practice, agents built in 2026 seldom lean on the bare window by itself. To last through long tasks they attach external memory: scratchpad files, a retrieval layer, or a knowledge base the agent keeps up. That does not contradict the point above, because external memory affects the model only by being read back into the window. It extends the buffer without replacing it, and the transformer's attention over that context remains the single place where thinking happens.
Where research is heading
Transformer performance follows predictable power laws in parameters, training data and compute, which gives researchers a roadmap. The frontier is moving along several lines:
- Efficient architectures that reduce the quadratic cost of attention, through Flash Attention, sparse attention and linear attention.
- Mixture of Experts, activating only the relevant parts of the network for each token.
- Better training data, where quality and diversity count as much as quantity.
- Multimodal transformers, with vision, audio, code and text in one architecture.
- Reasoning improvements, such as chain-of-thought, process reward models and test-time compute scaling, as seen in OpenAI's o-series and Anthropic's extended-thinking models.
All of these rest on the transformer foundation, so knowing it makes these directions readable.
The table below pulls every component into a single view, with the problem each solves, how it works and its main properties:
| Component | Problem it Solves | Core Mechanism | Key Properties |
|-----------|------------------|----------------|----------------|
| **Tokenization** | Text is not natively numeric | Subword BPE/WordPiece splitting | Vocabulary ≈ 30k–100k; subwords handle rare terms |
| **Embeddings** | Discrete IDs have no geometry | $\mathbf{e}_t = E[t] \in \mathbb{R}^{d_{\text{model}}}$ | Semantic similarity → geometric proximity |
| **Positional Encoding** | Parallel attention loses token order | sin/cos at different frequencies (or RoPE) | Multi-scale; encodes absolute and relative position |
| **Self-Attention** | Long-range deps, sequential bottleneck | $\text{softmax}(QK^\top/\sqrt{d_k})V$ | $O(n^2)$ but parallel; dynamic contextual reps |
| **Query/Key/Value** | Separating "what I seek" from "what I offer" | Three separate learned linear projections | Relevance ≠ similarity; flexible dependency modeling |
| **Multi-Head Attention** | Single head captures only one relationship type | $h$ parallel heads, concatenated + $W^O$ | Simultaneously captures syntactic, semantic, positional |
| **Encoder** | Understanding input in full bidirectional context | Full self-attention, no masking | Bidirectional; rich contextualized representations |
| **Decoder** | Generating output autoregressively | Causal self-attention + cross-attention | Predicts one token at a time; teacher-forcing in training |
| **Cross-Attention** | Connecting generation to input understanding | Q from decoder, K/V from encoder | Enables encoder→decoder communication |
| **Feed-Forward Network** | Per-token computation + factual storage | $\text{GeLU}(\mathbf{x}W_1)\,W_2$ per token | Position-wise; stores factual associations in weights |
| **Residual Connections** | Vanishing gradients in deep networks | $\mathbf{h} = \text{SubLayer}(\mathbf{x}) + \mathbf{x}$ | Creates gradient highway; enables 100+ layer models |
| **Pre-LayerNorm** | Unstable activations in deep networks | Normalize before each sublayer | Stable training; standard in all modern transformers |
What has been covered, and where to go next
The pieces fit together as follows. Parallel attention scales better than step-by-step processing and captures distant dependencies more effectively, which is why transformers displaced sequential models. Data flows from text to tokens to embeddings to positional encoding to attention to feed-forward layers, and then repeats. Self-attention is the core operation, with each token attending to every other through learned Q, K and V projections, as in Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V. Multiple heads capture varied patterns, positional encoding restores order, and the encoder-decoder design supports both understanding and generation. Feed-forward layers store knowledge, residual links allow depth and Pre-LayerNorm keeps training stable. Next-token prediction at scale yields emergent capability, the context window serves as an agent's working memory, and the architecture from a 2017 translation paper drives today's most advanced systems.
For going deeper, a staged plan works well.
First two weeks: the maths. Work through Andrej Karpathy's "Neural Networks: Zero to Hero" series on YouTube, especially the video that builds GPT from scratch. Then write a simple self-attention layer in NumPy. The concepts land differently once they have been computed by hand.
Weeks three and four: the papers. Read "Attention Is All You Need" (Vaswani et al., 2017), which is now readable, along with the BERT paper (Devlin et al., 2018), the GPT-3 paper "Language Models are Few-Shot Learners" (Brown et al., 2020) and "Scaling Laws for Neural Language Models" (Kaplan et al., 2020).
Month two: practice with libraries and APIs. The Hugging Face transformers library makes these models usable from Python with little setup. Fine-tune BERT or a small open-weight model such as a Llama or Qwen variant on a classification task. Then reach a current frontier model, whether Claude Sonnet 5 or one of the GPT-5 models, through the Anthropic or OpenAI Python SDK. Doing both ties theory to practice, because the same transformer principles are visible in open-weight and frontier API models alike.
Month three: one area in depth. Pick one of agent frameworks (LangChain, LlamaIndex, or hands-on time with Claude Code or Cursor to watch agentic behaviour in a real production tool), efficient transformers (Flash Attention, sparse and linear attention) or LLM fine-tuning with LoRA and PEFT. Depth beats breadth here.
Ongoing: keep up with research. The cs.CL and cs.LG sections of arXiv, the Hugging Face blog, Anthropic's research page and the AI Alignment Forum are good feeds. The field moves fast, but the foundation above makes new papers comprehensible.
The transformer paper runs about fifteen pages, and the architecture it describes is clean enough to implement in an afternoon. What it set loose, and what scaling and refinement keep setting loose, has changed what people think machines can do.