How a Transformer Reads "The Cat Sat on the Mat", Step by Step

A large language model is a deep learning model built on the transformer architecture and trained on huge amounts of text, so that it can understand and generate human language. Type a simple sentence into one, "The cat sat on the mat", and the model doesn't read it the way you do. It converts the words into numbers, attaches information about where each word sits, lets every word look at every other word to collect context, and then refines those numbers over and over until each word carries a richer, more meaningful representation.

All of that takes place inside a transformer, and the basic idea doesn't need heavy math. Just follow the sentence through, from raw text to a final batch of numbers that hold meaning, relationships and context. Below is the route "The cat sat on the mat" takes through one transformer layer.

Four ideas to hold on to

• Attention: tokens talk to each other.
• Feed-forward network (FFN): each token processes what it picked up.
• Residual connection: don't throw away the old information.
• Normalization: keep the numbers stable.

What goes into the block

1. Text

The input is the plain English sentence. Words don't turn into other words along the way. What changes at each step is the vector representation of each token.

2. Tokens

The sentence is chopped into small units called tokens, which are often whole words or pieces of words: "The", "cat", "sat", "on", "the", "mat".

3. Embeddings

Each token becomes a list of numbers, its embedding, which works as a rough first fingerprint of the word's meaning. At this point "cat" and "mat" already have different fingerprints, but neither knows anything about the rest of the sentence. Each token ID is turned into a vector:

                          The → [vector]
                          cat → [vector]
                          sat → [vector]
                          on → [vector]
                          the → [vector]
                          mat → [vector]

4. Position

Order matters, so each token also gets a marker for where it stands: first, second, and so on. "Cat" now knows it's word two and "mat" knows it's last. The fingerprints become a bit richer.

5. Self-attention

Here the transformer asks, for each token, which other tokens in the sentence are relevant. The model lets tokens look at one another and can learn strong links, such as "The" being tied to "cat", and "sat" describing an action that involves "cat". Each token looks at all the others and decides how much attention each deserves. "Sat" leans heavily on "cat", the one sitting, and on "mat", the place. The "the" in front of "mat" mostly attends to "mat".

After attention, every token has soaked up useful information from the rest of the sentence. "Cat" now has a vague sense that it's the one sitting, while "mat" has a hint that something is sitting on it. Put differently:

• cat becomes "cat plus information about its context"
• sat becomes "sat plus information about who is sitting"
• mat becomes "mat plus information about its relationship with 'on'"

The tokens have moved from isolated representations to context-aware ones.

6. Residual connection

The original token information shouldn't be discarded, so the attention result gets added to it. For "cat":

                           Original: "cat"
                             +Attention
              information: "cat is related to The and sat"
              Updated  : "cat + its contextual information"

7. Normalization

Next the representations are normalized:

                          Updated representations
                                  ↓
                               Normalize
                                  ↓
                          Well-scaled representations

Conceptually, for the sentence:

                            cat + context
                                  ↓
                       normalized cat + context


                           sat + context
                                  ↓
                        normalized sat + context

So what changed? The meaning isn't meant to change here. The numbers are rescaled so the network is more stable and easier to train.

8. Feed-forward network

Next, every token works on its own updated representation. A small neural network, the feed-forward network, examines each token's numbers separately and transforms them again, sharpening meanings, blending ideas or getting the vector ready for the next layer. "Cat" may come out with a firmer sense of being a sitting animal, and "mat" with a firmer sense of being a surface. This job isn't the same as attention's:

          In Attention "Which other tokens are useful?"
In FNN "Now that I have this information, how should I transform it?"

9. Second residual

The earlier information is preserved again, with the same idea applied to every token:

                          Before FFN:

                        "cat + context"

                               +

                          FFN output:

                      "refined representation"

                               ↓

                          After residual:

                "cat + context + refined information"

What changed:

                    Previous representation
                            +
                    FFN's new information
                            ↓
                    Richer representation

10. Second normalization

The updated representations are normalized once more, for stability:

                    Richer representations
                            ↓
                          Norm
                            ↓
                    Stable richer representations

11. A richer representation

What started as crude word fingerprints is now a much fuller set of numbers. Each token holds its own meaning, its position and how it relates to the other words. This fuller representation goes into the next transformer layer, or to the final output stage, where the cycle repeats and the model's reading of "The cat sat on the mat" deepens.

Before the block, each token holds mostly information about itself:

          The → mostly information about "The"

          cat → mostly information about "cat"

          sat → mostly information about "sat"

          on  → mostly information about "on"

          the → mostly information about "the"

          mat → mostly information about "mat"

After the block:

                            The
                             ↓
                 "The + information from its context"

                            cat
                             ↓
                 "cat + information about its relationships"

                            sat
                             ↓
                 "sat + information about cat and the sentence"

                            on
                             ↓
                 "on + relationship information"

                            the
                             ↓
                    "the + contextual information"

                            mat
                             ↓
              "mat + information about its relationship with on"

                             .
                             ↓
                   ". + sentence-ending context"

Then another transformer block follows:

        Block 1
     ──────────────

The → contextual representation
cat → contextual representation
sat → contextual representation
on  → contextual representation
the → contextual representation
mat → contextual representation


             ↓


          Block 2
      ──────────────

The → more refined representation
cat → more refined representation
sat → more refined representation
on  → more refined representation
the → more refined representation
mat → more refined representation

The same process then runs again across many layers. A transformer progressively transforms token representations, from simple token information to contextual information to highly rich representations the final model can use to predict the next token.

What it amounts to

A transformer doesn't understand language like a person. It takes a sentence such as "The cat sat on the mat" and splits it into tokens, turning each into numbers with position information added. Attention lets every word collect context from the others, residual connections and normalization blend and refine the numbers, and a feed-forward network transforms them further. After many repetitions the model holds a far richer numerical picture of the sentence, one that quietly encodes meaning, relationships and order. That picture is what lets large language models continue text, answer questions or come up with new ideas so fluently.