Why Transformers Use Sine and Cosine for Positional Encoding

Why do transformers use sine and cosine for positional encoding? Numbering the words 1, 2, 3 seems far simpler, yet that question is harder to answer properly than it sounds.

Attention has no sense of order

As an operation, attention treats its input like a bag of words. Give a transformer "cat eats mouse" and then "mouse eats cat", with no positional information at all, and the attention layer sees the same three vectors both times. It can't tell which came first, and the Q · K dot products come out identical. That's a real problem, since the first sentence is dinner and the second is a surprise. Position has to be injected somewhere before the first attention layer runs.

The obvious fix and why it fails

Take the phrase "cat and dog", three tokens with cat at position 1, "and" at position 2 and dog at position 3. Real word embeddings aren't two or three numbers long. Production transformers use 512, 768 or even larger vectors. Here are cat and dog at 768 dimensions, showing the first six entries and the last one to give you the shape:

cat ≈ [ 0.82, -0.14, 0.36, 0.05, -0.71, 0.29, ... , -0.18 ]   (768 numbers)
dog ≈ [ 0.77, -0.09, 0.41, 0.11, -0.68, 0.24, ... , -0.15 ]   (768 numbers)

The two vectors sit close together, which makes sense: both are animals, and embedding space puts words with similar meaning near each other. Yet nothing in either vector says where the word lives in a sentence. That's the gap.

The first thing most people try is adding the position number, as a plain scalar, to every entry of the embedding.

cat at position 1 → cat + 1 → [ 1.82, 0.86, 1.36, 1.05, 0.29, 1.29, ... , 0.82 ]

At position 1 it looks harmless. Now picture the same word at position 500 of a long document:

cat at position 500 → cat + 500 → [ 500.82, 499.86, 500.36, 500.05, 499.29, 500.29, ... , 499.82 ]

The first issue is the big one. The values balloon, and they do so identically in every dimension. What made cat recognizable, its particular run of 0.82, -0.14, 0.36 and so on, and what set it recognizably "cat" and different from "dog", turns into a rounding error next to 500. The signal is drowned out. Dividing by the maximum sequence length doesn't rescue it, because the maximum length isn't known when the model runs, and short sentences and very long documents would then sit on incompatible scales.

The second issue is subtler. Even before the blow-up, a single shared offset added to every dimension is a thin way to express position. The model has just one number to work with, which isn't enough surface to build relative-position behavior such as "attend to the word two places back".

So position should be a vector of its own, with the same shape as the word embedding so the two can simply be added, and it should stay well-behaved without needing outside facts like the sequence length.

Shrink the problem to see it clearly. Forget 768 dimensions and imagine only 2. Cat at position 1 needs some vector [x, y], and "and" at position 2 needs a different vector [p, q]. What should those numbers be? You could choose them by hand:

position 1 → [0.2, 0.8]
position 2 → [0.4, 0.6]
position 3 → [0.6, 0.4]

Every position does get a distinct pair, but it's arbitrary. Nothing links position 4 to position 3 the way they're linked on a number line. Ask for the vector at position 4,857 and there's no rule to apply; somebody would have had to pick it in advance, for every position anyone might ever use.

What's needed is a function. It takes an integer position and returns an [x, y] pair, puts nearby positions close together, and never lets the numbers explode, however large the position gets. Geometry solved this long ago. It's called a circle.

A clock for position

Put a point on a circle and let it move. Its coordinates are (cos θ, sin θ), so each angle yields a unique 2D coordinate, and neighboring angles yield neighboring coordinates. The representation is smooth. Instead of saying "position 17", you could write:

position 17 → (sin 17, cos 17)

That's two dimensions encoding one position, which already beats a scalar. Values stay within [-1, 1], so nothing blows up, and positions 17 and 18 look nearly alike.

One catch remains: a single clock repeats. After a full turn, position 1 and position 1 + 2π look the same. With a 500-token sentence and a clock whose period is about 6, collisions are everywhere.

Many clocks at different speeds

What makes the whole scheme work is running many clocks at once, each ticking at its own speed. One might cycle every 10 tokens or so, another every 100, another every 1,000, another every 10,000. For any position you record all the readings together. Two positions may agree on the fast clock, but they won't agree on every clock at the same time, so the combined reading acts like a fingerprint. GPS uses a similar trick with several satellites, and a barcode encodes an ID in stripes of different widths.

In the real formula, an embedding dimension d gives you d/2 clock pairs, with pair i running at frequency 1 / 10000^(2i/d). A small i is a fast clock and a large i a slow one:

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

That's the entire formula.

Worked example: "cat and dog" in 12 dimensions

Staring at the math in the abstract only goes so far, so compute it. With d = 12 there are 6 clock pairs, each with its own frequency, pair 1 the fastest and pair 6 the slowest:

pair 1: 10000^0/12  ≈ 1        (fastest)
pair 2: 10000^2/12  ≈ 21.54
pair 3: 10000^4/12  ≈ 464.16
pair 4: 10000^6/12  ≈ 1000
pair 5: 10000^8/12  ≈ 2154.43
pair 6: 10000^10/12 ≈ 4641.59  (slowest)

Take pair 1 and cat at position 1: the formula gives

A column is a word's full 12-dimensional PE vector, and the rows are where it gets interesting:

• Row 1 swings hard between words, going 0.84, then 0.91, then 0.14. The fast clock spins through most of a rotation in just two ticks.
• Row 6 hardly moves, showing 1.0000 three times running. The slow clock hasn't ticked in any meaningful way yet. It's held back for separating positions that sit far apart, such as 3 and 3000.
• Every entry is fully deterministic. Nothing in the table is learned.

The transformer's input is then a plain element-wise sum:

input = word_embedding(cat) + PE(1)

Same shape, no fancy fusion. To see the actual output, the word embeddings are shrunk to the toy PE's 12 dimensions (a real model would use its true 768-dim vectors, and these numbers are made up purely to show the mechanism):

cat_embedding ≈ [ 0.20,  0.70, -0.10,  0.40,  0.15, -0.30,  0.55,  0.10, -0.20,  0.05,  0.30, -0.15]
dog_embedding ≈ [ 0.18,  0.68, -0.05,  0.35,  0.20, -0.25,  0.50,  0.15, -0.18,  0.08,  0.28, -0.10]

As with the 768-dim versions, cat and dog start out close because they're both animals. Now add each word's PE from the table, dimension by dimension:

cat_final = cat_embedding + PE(1)
          = [1.0415, 1.2403, -0.0536, 1.3989, 0.1522, 0.7000, 0.5510, 1.1000, -0.1995, 1.0500, 0.3002, 0.8500]

dog_final = dog_embedding + PE(3)
          = [0.3211, -0.3100, 0.0888, 1.3403, 0.2065, 0.7500,
             0.5030, 1.1500, -0.1786, 1.0800, 0.2806, 0.9000]

Those two vectors are the ones that actually reach the first attention layer. Compare dimension 2: cat and dog were nearly the same there (0.70 against 0.68), and after the positional signal they're at 1.24 versus -0.31, worlds apart. That's the fast clock (pair 1, position 1 versus position 3) working as intended. It leaves enough of a positional stamp that "cat, early in the sentence" and "dog, later in the sentence" become clearly different vectors although the words themselves mean similar things.

How does the model know [0.84, 0.54, ...] means position 1?

It doesn't. No dimension in that vector says "1", and nothing decodes it. The model never has to recover the position number, because that isn't its job.

The job is narrower. Suppose dog's query meets cat's key in a dot product. Will the score show that the two words are two slots apart? That's the whole meaning of "understanding position" at this layer. The encoding isn't noise. It's a smooth mix of sinusoids at known frequencies, so the learned Q and K projections can pick up on those patterns and respond to them. A query can end up meaning "favor whichever key is one place earlier", and no variable called position is ever involved.

Face recognition offers a handy comparison. A 512-dimensional face embedding has no single dimension labeled with the person's name. The whole vector, taken together, is that person. The 12-dim PE(1) vector works the same way: collectively, it is position 1.

Why both sine and cosine

This is the part that usually gets hand-waved. The real reason is that the pair together gives a representation of position that's friendly to rotation. The angle-addition identities say:

sin(x + Δ) = sin(x)cos(Δ) + cos(x)sin(Δ)
cos(x + Δ) = cos(x)cos(Δ) − sin(x)sin(Δ)

Know [sin(x), cos(x)] and you can compute [sin(x+Δ), cos(x+Δ)] through a fixed linear transformation that depends on Δ only and not on x. Shifting position by Δ is therefore the same transformation wherever you start. That lets attention learn "look Δ tokens back" as one reusable operation, with no need to memorize a separate rule for each pair of positions. Sine alone can't do this, because you need both halves of the pair to pin down the full rotation.

What happens in backprop

During training, are the positional encodings adjusted by gradient descent? For the sinusoidal kind, no. They have no learnable parameters, and the formula is fixed. Gradients flow through the sum (input = embedding + PE) without finding anything to update on the PE side, much like asking whether the 5 in y = x + 5 gets trained.

The word embedding table is different. Each time cat shows up in a batch, its row in the table receives a gradient and shifts.

One more variant deserves a mention, because most modern models actually use it: learned positions. There's no formula at all. The model keeps a trainable table with one row per position, up to the maximum sequence length and as wide as the model dimension, and training decides what row 17 should hold. That's easier to reason about and works well within the lengths seen in training, but it can't stretch to longer sequences the way sinusoidal PE can. That trade-off, along with newer approaches such as ALiBi and RoPE, is a topic of its own.