How Embedding Layers Work: From Text to Token IDs to Vectors in an LLM

Type the sentence "I love cats" into an LLM and the model never sees three English words. Neural networks operate on numbers. So how does that short phrase turn into something a Transformer can process? The answer begins with the embedding layer.

1. From text to token IDs

Tokenization comes first. A tokenizer splits the text into pieces called tokens and gives each piece an ID. To keep things simple, assume this tiny vocabulary:

"I"     → 10
"love"  → 25
"cats"  → 47

The sentence then becomes a short list of integers:

"I love cats"
      ↓
[10, 25, 47]

Those are token IDs, and they carry no meaning by themselves. The number 47 doesn't say that "cats" is worth forty-seven of anything. It's only an index, a label that points at one entry in the vocabulary. One more step is needed before the model can do anything useful.

2. From token IDs to embeddings

That step belongs to the embedding layer. Picture it as a big table with one row, which is a vector, for every token in the vocabulary.

Say the vocabulary has just 50 tokens and each token is described by 4 numbers. Each row is an embedding vector:

Embedding vector

The table might begin like this:

Token 0    →    [ 0.2, -0.4,  0.7,  0.1 ]
Token 1    →    [ 0.6,  0.1, -0.3,  0.8 ]
...
Token 10   →    [ 0.3, -0.2,  0.9,  0.4 ]
...
Token 25   →    [-0.5,  0.7,  0.2, -0.1 ]
...
Token 47   →    [ 0.8,  0.1, -0.6,  0.5 ]
...

Now take the input sequence:

[10, 25, 47]

The layer simply looks up those rows. Token 10 gets its vector:

10 → [ 0.3, -0.2,  0.9,  0.4 ]

and the other two follow the same way:

25 → [-0.5,  0.7,  0.2, -0.1 ]47 → [ 0.8,  0.1, -0.6,  0.5 ]

So the original text goes through these stages:

"I love cats"

      ↓ Tokenizer[10, 25, 47]      ↓ Embedding lookup[
 [ 0.3, -0.2,  0.9,  0.4 ],
 [-0.5,  0.7,  0.2, -0.1 ],
 [ 0.8,  0.1, -0.6,  0.5 ]
]      ↓Transformer

That's the job in a nutshell: the embedding layer turns token IDs into vectors.

3. Why vectors and not raw IDs?

You might ask why the Transformer can't just receive the IDs directly. The reason is that IDs are only labels. Look at the three values:

10
25
47

The gap between 25 and 47 says nothing about how "love" relates to "cats". Embedding vectors are different because they're learned representations. During training, the model adjusts their values until they help the Transformer handle language.

4. Where the numbers come from

Nobody types the values of the embedding matrix by hand. They're learned during training. At the start, the vectors hold values with no linguistic information in them. The model then reads huge amounts of text and practices predicting the next token. Given a prompt like "I love cats because they are ___", it makes a guess.

When the guess is wrong, the error flows backward through the network by backpropagation. The embedding vectors get updated together with every other weight. After training, the matrix holds representations the model has learned to rely on.

5. What an embedding represents

Take a vector for "cats":

cats → [0.8, 0.1, -0.6, 0.5]

It's tempting to read each number as a named property, like this:

0.8 → animal
0.1 → size
-0.6 → shape
0.5 → plural

In general, LLM embeddings don't work that way. Meaning is spread across all the dimensions at once. One dimension rarely matches a single concept a person could name. The whole vector is the representation.

6. Real embedding matrices are huge

Now scale up from the toy example to a realistic model with these dimensions:

Vocabulary size = 50,000
Embedding dimension = 4,096

The embedding matrix then has about 205 million parameters, which is a lot. What looks like a plain lookup table can hold a surprisingly large share of a model's weights.

7. What happens next

Once "I love cats" has become a set of embedding vectors, they go into the Transformer. Here's the flow:

"I love cats"
               │
               ▼
          Tokenizer
               │
               ▼
        [10, 25, 47]
               │
               ▼
       Embedding Layer
               │
               ▼
      [vector, vector, vector]
               │
               ▼
          Transformer
               │
               ▼
       Next-token prediction

The embedding is the bridge between discrete tokens and the continuous numerical representations that the neural network works with.

8. The same idea in Python

A few lines of PyTorch are enough to show it:

import torch
import torch.nn as nn
# Vocabulary contains 50 tokens
# Each token is represented by 4 numbers
embedding = nn.Embedding(
    num_embeddings=50,
    embedding_dim=4
)
# Token IDs for "I love cats"
tokens = torch.tensor([10, 25, 47])
# Look up their embeddings
vectors = embedding(tokens)
print(vectors)
print(vectors.shape)

The printed result looks roughly like this:

tensor([
    [ 0.31, -0.21,  0.91,  0.42],
    [-0.52,  0.73,  0.19, -0.12],
    [ 0.79,  0.08, -0.61,  0.54]
])

And the shape confirms three tokens with four numbers each:

torch.Size([3, 4])

The embedding layer is basically a learned lookup table. Put the pieces together and the pipeline looks like this:

Text
  ↓
Tokenizer
  ↓
Token IDs
  ↓
Embedding lookup
  ↓
Vectors

Transformer

The tokenizer identifies which tokens are present. The embedding layer converts their IDs into learned numerical representations. The Transformer takes those representations, works out the context and makes predictions.

Next time you meet this line in a model definition:

nn.Embedding(vocab_size, embedding_dim)

read it as an instruction to build a table with vocab_size rows, where each token receives its own learnable vector of length embedding_dim.