Residual Connections Explained: Why Deep Transformers Can Train at All

Stack dozens of layers naively and training turns unstable. Residual connections, also called shortcuts, fix that by letting both information and gradients travel straight through the network.

Why plain stacking fails

By this point a Transformer block has three parts: multi-head self-attention, layer normalization and a feed-forward network. It's tempting to repeat them one after another:

Input
   │
   ▼
Attention
   │
   ▼
Feed Forward
   │
   ▼
Attention
   │
   ▼
Feed Forward
   │
   ▼
...

That doesn't train well. The deeper a network gets, the harder it is to optimize, and this was among the biggest obstacles in deep learning until residual connections arrived with ResNet in 2015. Almost every current Transformer, GPT, BERT, LLaMA, Claude and Gemini included, depends on them.

Start with a network of two layers:

Input  ↓Layer 1  ↓Layer 2  ↓Output

Training it is usually painless. Now picture 96 layers:

Input↓Layer 1↓Layer 2↓...↓Layer 96↓Output

Each layer transforms the output of the one before. During backpropagation, the gradient has to pass back through every layer on its way down, and on the trip it can shrink toward nothing (vanishing), grow without bound (exploding) or get noisier and noisier. In the end the earliest layers barely learn at all.

The whisper chain

Think of a message whispered along fifty people:

Person 1↓Person 2↓Person 3↓...↓Person 50

When it reaches the end, it has changed a lot. Parts are missing and parts are garbled. A deep network behaves similarly, since each layer nudges the representation a little, and after enough of them the useful signal can fade.

Add a shortcut

What if the data didn't have to go through every layer and could also skip around them?

Input
  │
  ├──────────────┐
  ▼              │
Layer            │
  ▼              │
Output           │
  │              │
  └────── Add ◄──┘
        │
        ▼
Final Output

The original input now always has a direct route to later layers, and that small change alters how deep networks behave in a big way.

The math

Say a layer computes this:

F(x)

Without a residual connection, the output is just the layer's result:

Output = F(x)

With one, the output becomes:

Output = x + F(x)

The input is added straight onto the transformed output. This operation is known as a skip connection or shortcut connection.

Why it helps

Suppose a given layer has nothing useful to contribute. In a plain network, it would somehow have to learn to act as an identity function, which turns out to be hard. With a residual connection, the identity comes for free. The layer only needs to learn:

F(x) = 0

so that:

Output = x + 0 = x

The input goes through untouched, and optimization gets far easier.

Learning the difference

This is where the name comes from. A layer doesn't have to learn the whole transformation. It learns only the difference, the residual, between the input and the target output. Photo editing is a good comparison: you rarely repaint the entire picture, you adjust it by brightening, raising contrast or sharpening edges. A Transformer layer likewise seldom needs to replace the whole representation. It just has to improve it slightly.

Two shortcuts per block

Each Transformer block holds two residual connections. The first wraps the attention layer:

Input
  │
  ├──────────────┐
  ▼              │
Multi-Head Attention
  ▼              │
 Add ◄───────────┘
  │
  ▼
Output

The second wraps the feed-forward network:

Input
  │
  ├──────────────┐
  ▼              │
Feed Forward Network
  ▼              │
 Add ◄───────────┘
  │
  ▼
Output

Pre-LayerNorm

Current GPT models use the Pre-LayerNorm layout, computed like this:

Input
   │
   ▼
LayerNorm
   │
   ▼
Multi-Head Attention
   │
   ▼
Add Residual
   │
   ▼
LayerNorm
   │
   ▼
Feed Forward Network
   │
   ▼
Add Residual
   │
   ▼
Output

LayerNorm runs before each sub-layer here, not after, and that arrangement makes deep Transformers noticeably more stable to train.

Gradient flow

In backpropagation, the gradient now has two routes:

Gradient↓Transformer Layer↓Residual Path

If the main transformation turns out hard to optimize, the shortcut still offers another way back, so the gradient can't disappear completely. In terms of calculus, the derivative of

x + F(x)

is

1 + F'(x)

That constant 1 matters a great deal. Even if

F'(x)

gets very small, the gradient still has a direct path through the identity link. This is among the main reasons residual networks train successfully.

A numeric example

Say the input embedding is:

[0.5, 1.2, -0.3]

and attention outputs:

[0.1, -0.2, 0.4]

Rather than replacing the input, the two are added:

[0.5, 1.2, -0.3] + [0.1, -0.2, 0.4] = [0.6, 1.0, 0.1]

The original information survives while the new information is mixed in.

In PyTorch

The code cost is almost nothing:

x = x + self.attention(
    self.layer_norm1(x)
)

x = x + self.ffn(
    self.layer_norm2(x)
)

That addition is one of the most significant operations in the whole architecture.

One full block

                Input
                  │
                  ▼
             LayerNorm
                  │
                  ▼
        Multi-Head Attention
                  │
                  ▼
           Add Residual
                  │
                  ▼
             LayerNorm
                  │
                  ▼
      Feed-Forward Network
                  │
                  ▼
           Add Residual
                  │
                  ▼
                Output

Modern LLMs repeat this structure dozens or even hundreds of times.

What changes with and without

Lacking residual connections, deep networks are hard to optimize, gradients vanish more readily, information gradually drains away, and extra layers often make results worse. With them, information moves directly through the network, gradients stay stable, deeper models become trainable and optimization gets much easier. Residual learning is one of the breakthroughs that made today's large language models possible.