From Prompt to Token: How an LLM Turns Text Into Numbers and Back

No English inside the model

Hit Enter on a prompt and a GPU starts working through billions of floating-point operations before the first token shows up. Not one of those operations touches an English word.

Take a familiar fragment:

"The cat sat on the "

By the time "mat" appears on screen, that fragment has been taken apart, turned into geometry with thousands of dimensions, and pushed through dozens of identical mathematical layers.

We talk about this loosely. The model "read" the prompt, "understood" the context and "chose" a sensible continuation. None of that literally happens. The running program contains no English, no grammar rules splitting subject from predicate, and no stored dictionary definitions. What happens is an assembly line of tensor operations: a string becomes integers, the integers become coordinates in space, the coordinates get mixed across dozens of layers, and at the end a weighted die gets rolled once.

To really understand language models, drop the metaphors and follow the data. Here is the route your text takes.

Step 1: the tokenizer

Neural networks can't work on raw characters, because there's no algebra to do on the letter "c". They need numbers. So before the prompt gets anywhere near the model, a separate program called the tokenizer processes it.

The tokenizer cuts the string into pieces called tokens. A token might be a whole word, a syllable, a prefix or a single punctuation mark. Our phrase splits cleanly into five:

• "The"
• " cat"
• " sat"
• " on"
• " the"

Look at the leading spaces. They aren't typos. Modern tokenizers glue the whitespace in front of a word onto the token, which makes "cat" and " cat" two different tokens.

Every token maps to one integer in a fixed vocabulary table. With a vocabulary of, say, 50257 tokens, the phrase turns into five numbers:

[464, 3797, 3372, 319, 262]

No layer of the language model has run yet, and the string has already vanished. Five integers are all that remain.

Step 2: the embedding table

Now the network proper takes over, and its first job is to fix a basic problem. Integers are useless for calculus. The number 3797, which stands for " cat", is no closer to "dog" than to "airplane". These are arbitrary labels, and you can't do geometry with labels.

The fix is to turn each integer into a dense list of floats, an embedding vector. Picture a giant lookup table sitting in GPU memory. Suppose the vocabulary holds 32,000 tokens and the hidden dimension is 4,096. The table is then a matrix of 32,000 rows by 4,096 columns, and the model simply looks things up:

• Token 464 fetches row 464.
• Token 3797 fetches row 3797.
• Token 3372 fetches row 3372.
• Token 319 fetches row 319.
• Token 262 fetches row 262.

No arithmetic is involved at this stage. It's a plain memory read. What comes out is a matrix of shape [5, 4096], five rows of 4,096 continuous values each.

Every token is now a point in a 4,096-dimensional space where direction carries meaning, and where positions encode relationships learned from billions of training examples.

One more ingredient goes in before the vectors descend further: positional information. Matrix operations by themselves can scramble word order, so the model mixes position signals into the coordinates. Without that, "The cat sat on the" and "The on sat cat the" would look identical to the network.

Step 3: the transformer stack

The [5, 4096] matrix now enters the transformer. In a typical 7B-parameter design the stack is 32 identical blocks. The matrix keeps its shape through every one of them, going into layer 1 and coming out of layer 32 as [5, 4096]. What changes is the information stored in those numbers.

Straight out of the embedding table, each vector sits alone. The vector for " sat" knows it means sitting, but it can't tell who sat, where, or why. Two mechanisms inside each block fix that.

Attention

Here the vectors communicate. Attention lets every position look back at earlier positions and pull out what's relevant.

Take the fifth token, " the". It measures similarity against "The", " cat", " sat" and " on". Since it has to help predict which noun follows a preposition, its vector draws strongly on the action and on the subject. Through dot products across these positions, the attention layer refreshes each vector with context from its neighbors.

The feed-forward network

After the tokens trade information, each row goes through a feed-forward network (an MLP) on its own. If attention is tokens talking to each other, this layer is each token doing its own math. It projects the freshly gathered context through thousands of learned weights, which lets the model switch on factual associations stored in its parameters.

All of this repeats 32 times. By layer 32, the fifth vector is no longer a generic stand-in for the word " the", since it has absorbed the history of the whole sentence. It now stands for a definite article that follows a sitting cat.

Unembedding

To produce the next word, the first four rows no longer matter, so the model takes just the final row: one vector of 4,096 floats. Now the embedding problem runs in reverse. It holds a point in 4,096-dimensional space and needs a word.

A second matrix, the unembedding matrix (frequently just the embedding table transposed), multiplies that vector and spreads its 4,096 values back out over the whole vocabulary. The result is one unnormalized score for each token in the vocabulary, for example:

Near the top you'd see " mat" at 16.2 and " rug" at 13.8, followed by " floor" at 12.1 and " couch" at 10.4. Far down the list, "LeBron James" sits at -12.5.

These raw scores are logits. A higher logit means the network's learned parameters found a stronger geometric match between the final state of the prompt and that candidate.

Softmax

Logits can't be used as they are. A few are large and positive, others are negative, and their total is arbitrary. So the model runs the entire vector through softmax: for each entry, take e raised to its score, then divide by the total of those exponentials across every entry.

Softmax does two jobs. Taking the exponential makes every value strictly positive. Dividing by the total then forces everything to sum to 1.0. What you get is a true probability distribution covering all 32,000 tokens, roughly:

The leader, " mat", takes 68.4%. " rug" gets 17.1%, " floor" 9.2% and " couch" 4.1%. At the far end, " LeBron James" is left with 0.000001%.

Sampling

Everything up to this point is deterministic. With identical weights and identical input, the probabilities won't differ by even one decimal place. The unpredictability of LLMs enters at the last step, sampling.

If the system always grabbed the most probable token, here " mat" at 68.4%, that would be greedy decoding. It's deterministic, but over longer outputs it tends to fall into repetitive, unnatural loops. So the software rolls a weighted die instead. About 68% of the time it lands on " mat", and about 17% of the time on " rug".

The temperature setting in an API call reshapes this curve before the die is rolled. A low temperature exaggerates the gaps and might push " mat" toward 95%. A high temperature flattens the curve, so lower-ranked tokens get a much better shot at being picked.

The die rolls, and the winner is token ID 2603.

Detokenization and the loop

The program looks up ID 2603 in a reverse vocabulary map, finds the string " mat", and sends it to your terminal or browser.

This is where the central fact of inference shows up: at no point does the model know what comes after " mat". It can't plan three words ahead and keeps no outline of the paragraph it's writing. To get further it appends the new word to the input and forms a brand-new prompt:

"The cat sat on the mat"

It tokenizes all six words, embeds all six rows, sends all six positions through all 32 layers, computes the logits for position six, applies softmax, rolls the die and picks the seventh token. That full cycle, from string to tensor to die roll, repeats once for every token that appears on your screen.

Questions the pipeline raises

Looked at as an engineer, the mechanics provoke some awkward questions. Each generated word seems to require pushing the entire accumulated conversation through billions of operations, so long chats should bring the hardware to a crawl. (Why they don't comes down to memory caching, a topic for another day.)

The first step is even more puzzling: the jump from human language to numbers. Before attention fired or any layer ran, the prompt was already chopped into arbitrary numeric chunks. Why subwords, and not characters, whole words or raw bytes? Why do unusual strings or lines of Python make token counts behave unpredictably? Tokenization is the first box in the chain, and it's the one to open next.