Putting it together · Years 10–12 · 75 minutes

Inside a language model: one sentence, start to finish

What happens when you type “The world is really changing because of the rise and fast adoption of Artificial”?

Follow one sentence through a real language model, GPT-2 small, to its next word. It is cut into tokens, turned into vectors, stamped with positions, mixed by attention, reworked by MLPs twelve times over, and finally turned back into scores and probabilities. Every step is maths from the first three lessons.

your text
1 · Tokens
2 · Embeddings
3 · Positions
4 · Attention
5 · MLP (×12 with attention)
6 · Unembedding
7 · Logits
8 · Softmax
next token

Where this happens: The whole forward pass, from typed text to the next token.

By the end you can

Describe, in order, what happens to a sentence inside a transformer language model.
Explain how dot products, matrices and softmax appear inside attention, MLPs and the output.
Use real measurements to show where the next word emerges, and what the model’s probabilities do and do not mean.

Starting knowledge: Lessons 1 and 3 (vectors, dot products, softmax). Lesson 2 helps with the idea that training shaped every number.

Watch

See the idea move

The animations are silent; captions carry the explanation. Pause whenever a caption changes and say what you expect next.

0:00 / 0:28

One sentence, start to finish

0:28 · no sound

The whole trip in half a minute: tokens, vectors, positions, attention, twelve blocks, logits and softmax, ending with “Intelligence” at 98%.

What happens on screen

The sentence is cut into 14 tokens.
Each token becomes a vector, and a position vector is added.
Attention links “Artificial” back to earlier tokens such as “adoption”.
Twelve blocks of attention and MLP rework the vectors.
The last vector is scored against every token; softmax gives “Intelligence” 98.0%.
The chosen token joins the sentence and the trip starts again.

Predict, then test

The journey of one sentence

Eight stages, each with a short explanation, an animation or a hands-on panel, and a question to think about. Go in order the first time.

Explain

Explain what you found

Tell the story of the token “Artificial” in eight short steps, from its ID to the prediction.
At which block does GPT-2’s top guess become “Intelligence”, and was it attention or the MLP that made the jump? What was its guess before that?
With the typo “Artifical”, GPT-2’s top guess drops to “.” at 4%. Which stage caused the trouble, and why can later stages not simply fix it?

Counterexample challenge. This lesson simplifies a lot: the animations show 16 of 768 numbers, 3 of 144 attention heads and one MLP. Choose one simplification and explain what a learner might wrongly conclude from it.

Teacher notes

Teaching this lesson

Suggested sequence · 75 minutes

Warm-up (5 min): write the sentence on the board and ask for the next word. How sure is the class?
Watch “One sentence, start to finish” as the map for the lesson.
Prediction, then stages 1–3 (20 min): tokens, embeddings, positions. Pairs try every tokenisation variant.
Stages 4–5 (20 min): attention and the stack of blocks. Record the block where “Intelligence” takes over.
Stages 6–8 (15 min): unembedding, logits and softmax. Compare samples at temperature 1 and 2.
Exit task (15 min): the explanation prompts, or the counterexample challenge for fast finishers.

Evidence to collect

An ordered, accurate description of the eight stages; one claim supported by a real measurement from the lesson (for example the block-7 jump); a clear statement of what a 97.95% probability does and does not mean.

Limits of this lesson

Real measurements from GPT-2 small (124 million parameters, 2019): tokens and IDs, embedding neighbours and the 2-D map, position waves, the three attention heads, block-by-block guesses, logits and probabilities. Illustrations: the embedding-table cells, the MLP neuron grid, the value-mixing strips and the attention score bars (derived from real weights). Block-by-block guesses decode an unfinished vector with the model’s final layer: a useful approximation, not what the model computes at that point. Modern models are far larger and differ in details such as rotary positions, but follow the same pipeline.

Common misconceptions

“The model looks the answer up.” No sentence is stored and retrieved; scores are computed from learned numbers.
“Attention means the model understands like a person.” It is a weighted average steered by dot products. Some heads look useful; many rest on the first token.
“It predicts words.” It predicts tokens. “Intelligence” is one token; “Artifical” is three.
“97.95% means it is true.” It means the continuation is typical of the training text.

Glossary

token: A piece of text from a fixed list; the model sees only its number.
embedding: The learned vector a token starts with: 768 numbers in GPT-2 small.
position vector: A learned vector added to mark where a token is.
attention: Each token takes a weighted mix of earlier tokens, with weights from query · key and softmax.
head: One attention pattern. GPT-2 small has 12 heads in each of 12 blocks.
MLP: A layer, a bend and a layer applied to each token on its own.
block: Attention followed by an MLP. GPT-2 small stacks 12.
unembedding: Dot products between the final vector and each token’s output vector.
logit: The raw score for one possible next token.
softmax: Turns logits into probabilities that add to 1.

Related Brightlab investigations

Brightlab

Opening your investigation…

Inside a language model: one sentence, start to finish

What happens when you type “The world is really changing because of the rise and fast adoption of Artificial”?

By the end you can

Describe, in order, what happens to a sentence inside a transformer language model.

Explain how dot products, matrices and softmax appear inside attention, MLPs and the output.

Use real measurements to show where the next word emerges, and what the model’s probabilities do and do not mean.

Starting knowledge: Lessons 1 and 3 (vectors, dot products, softmax). Lesson 2 helps with the idea that training shaped every number.

See the idea move

The animations are silent; captions carry the explanation. Pause whenever a caption changes and say what you expect next.

0:00 / 0:28

One sentence, start to finish

0:28 · no sound

The whole trip in half a minute: tokens, vectors, positions, attention, twelve blocks, logits and softmax, ending with “Intelligence” at 98%.

What happens on screen

The sentence is cut into 14 tokens.
Each token becomes a vector, and a position vector is added.
Attention links “Artificial” back to earlier tokens such as “adoption”.
Twelve blocks of attention and MLP rework the vectors.
The last vector is scored against every token; softmax gives “Intelligence” 98.0%.
The chosen token joins the sentence and the trip starts again.

The journey of one sentence

Eight stages, each with a short explanation, an animation or a hands-on panel, and a question to think about. Go in order the first time.

Explain what you found

Tell the story of the token “Artificial” in eight short steps, from its ID to the prediction.

At which block does GPT-2’s top guess become “Intelligence”, and was it attention or the MLP that made the jump? What was its guess before that?

With the typo “Artifical”, GPT-2’s top guess drops to “.” at 4%. Which stage caused the trouble, and why can later stages not simply fix it?

Teaching this lesson

Suggested sequence · 75 minutes

Warm-up (5 min): write the sentence on the board and ask for the next word. How sure is the class?
Watch “One sentence, start to finish” as the map for the lesson.
Prediction, then stages 1–3 (20 min): tokens, embeddings, positions. Pairs try every tokenisation variant.
Stages 4–5 (20 min): attention and the stack of blocks. Record the block where “Intelligence” takes over.
Stages 6–8 (15 min): unembedding, logits and softmax. Compare samples at temperature 1 and 2.
Exit task (15 min): the explanation prompts, or the counterexample challenge for fast finishers.

Evidence to collect

Limits of this lesson

Common misconceptions

“The model looks the answer up.” No sentence is stored and retrieved; scores are computed from learned numbers.
“Attention means the model understands like a person.” It is a weighted average steered by dot products. Some heads look useful; many rest on the first token.
“It predicts words.” It predicts tokens. “Intelligence” is one token; “Artifical” is three.
“97.95% means it is true.” It means the continuation is typical of the training text.

Glossary

token: A piece of text from a fixed list; the model sees only its number.
embedding: The learned vector a token starts with: 768 numbers in GPT-2 small.
position vector: A learned vector added to mark where a token is.
attention: Each token takes a weighted mix of earlier tokens, with weights from query · key and softmax.
head: One attention pattern. GPT-2 small has 12 heads in each of 12 blocks.
MLP: A layer, a bend and a layer applied to each token on its own.
block: Attention followed by an MLP. GPT-2 small stacks 12.
unembedding: Dot products between the final vector and each token’s output vector.
logit: The raw score for one possible next token.
softmax: Turns logits into probabilities that add to 1.