Brightlab
brightlab.Made for curious mindsbrightlab.Made for curious mindsBrightlab
How does a network turn numbers into a confident—or unsure—guess?
A language model’s last layer gives a score to every possible next word. Softmax turns the scores into probabilities. Cross-entropy measures how surprised the model was by the right answer, and sampling turns probabilities into an actual choice.
Where this happens: The output: scores become a guess, and a guess becomes a loss.
Starting knowledge: Fractions, percentages and probabilities that add to 1. The exponential eˣ and natural logarithm are introduced as tools.
The animations are silent; captions carry the explanation. Pause whenever a caption changes and say what you expect next.
Four candidate next words for “The cat sat on the …”. Their scores become positive with eᶻ, then are divided by the total so they add to 1. Temperature sharpens or flattens the result.
Cross-entropy loss is −ln of the probability given to the right answer, so confident mistakes cost the most. Its gradient, p − target, says which scores training should raise and lower.
Set the four scores and the temperature. The lab shows eᶻ, the probabilities, the loss for the right answer and the gradient. Then sample words and compare the counts with the probabilities.
A random number between 0 and 1 lands in one slice of the probability strip. A few samples are noisy; after 1,000 the counts are close to the probabilities.
Counterexample challenge. Make “moon” the most likely word by changing only the temperature. If you cannot, explain why not.
Softmax probabilities calculated for three scores; an explanation of shift invariance supported by lab evidence; a comparison of the loss for confident-right, unsure and confident-wrong guesses.
The four words and their scores are an illustrative example, not output from a real language model; real models score tens of thousands of tokens. Sampling in the lab uses a seeded generator so a run can be repeated.