brightlab.Model detective / Year 6
Can you catch a model taking a wrong turn?
A language model can get the answer right for a reason we do not understand — or confidently get it wrong. Your job: gather evidence, look inside, and test an explanation.
One connected investigation. Real calculations from a small trained model. No coding needed.
01 / Observe before explaining
Give it a case to solve.
Change the characters and their bags. The facts tell you the answer. Will the model agree? Its answers come from learned weights, not an “if Pip, say red” rule.
We keep the two colours different so the swapping experiment later has something to change.
A prediction is a starting idea, not a test to pass.
The model’s answer will appear here.
Make a prediction, then run the case. You can inspect every possible next word, including unlikely ones.
Find a case where the model is very confident and wrong. What does that tell you about confidence?
Probabilities cover all ten words in its vocabulary. It was trained to predict the final colour, not to write arbitrary text.
The Goodfire connection: learn from behaviour, then look inside
Mixing Mechanisms studies how models retrieve facts attached to particular names. This small transformer has a related task. Similar tasks do not mean that two models use the same mechanism.
02 / A bright signal is a clue
Follow a connection. Then cut it.
An attention head mixes information from earlier words. The model works through two stages, called blocks. Each block contains two heads. Pick a head and a word position to see its actual attention weights. Thick lines mean larger weights — not proven causes.
Run a case above to open the model.
Which earlier words does this position attend to?
Numbers below the diagram provide the same evidence as line thickness. A position cannot look ahead. This chart is from the intact model.
Disabling sets this head’s output to zero at every position. Other heads and both blocks still run. It is a fresh intervention, not a permanent edit.
Hunt for an internal pattern.
These are the 48 neuron activations after the attention step in your selected block, at the final word position. Pick a square to compare it across six cases.
Darker green = larger activation. A neuron is one unit in the model. It is not automatically a neatly labelled idea.
What sets this neuron off?
| Case | Activation | Model says |
|---|
Does it react to a name, a colour, or a combination? Compare the rows. Then make a new case above that could prove your explanation wrong.
The Goodfire connection: features and circuits
Goodfire uses sparse autoencoders to recover patterns from mixed model signals. Those patterns can combine many neurons. Our view exposes raw neurons; it does not pretend each square is a recovered feature.
Circuit-tracing research tests proposed paths through model computations. Cutting a head is one test, with limits: a change might disrupt several useful computations at once.
03 / Change the inside, keep the input fixed
Can a piece of one run change another?
We make a second prompt by swapping the bag colours. Then we transplant one 24-number internal state from the original run into the swapped run. The input stays swapped. Only that internal state changes.
Here, “memory” means an intermediate list of numbers, not a person’s memory. Your aim is to see whether a transplant restores the original answer. That original answer may now disagree with the swapped facts.
Run a case above to prepare the two runs.
Donor: the original run
Read the 24 donor numbers
Recipient: colours swapped
Read the 24 recipient numbers
| Run | Answer | Original-answer probability |
|---|
Try the final position after Block 2, then an earlier position. Use the recipient’s own vector as a control. Which changes matter? A final-state transplant is a useful check, but it does not locate where the answer was first worked out.
The Goodfire connection: causal intervention
Mixing Mechanisms and the circuits research landscape investigate mechanisms behind model behaviour. We execute a real residual-state patch in this tiny transformer. Our result is evidence about this computation, not a reproduction of a frontier-model finding.
04 / Compare models, not just answers
Something changed during training.
We have a second version of the same model. It had extra training on deliberately changed labels. Before revealing what changed, look for a pattern in who gets helped or harmed.
Run a case above to begin the comparison.
Your current case, two checkpoints
| Version | Answer | Probability of factual colour |
|---|
Both versions run on your current prompt. Change the characters above and collect more than one example.
| Asked about | Original correct | Changed correct | Cases |
|---|
Open the training record
The changed model received 70 extra training steps. In those extra examples, answer labels for queries about Kiki were replaced with “green”. The model learned from those labels. It did not receive a hand-written rule saying “if Kiki, output green”.
This is an intentionally introduced training bias in fictional data. Results for four names and four colours do not establish how a model treats real people.
These 58 combinations were reserved from the original training set. After you inspect them, they are no longer an unseen test for your own developing explanation. A fresh evaluation would need new reserved cases.
The Goodfire connection: model differences
Model Diff Amplification searches for behaviours that differ between model versions. We directly compare two checkpoints here; we do not amplify their differences. Looking for a change is different from measuring how often it happens in ordinary use.
Your evidence board
Make a claim you can defend.
Pin runs as you explore. Then explain the strongest thing your evidence supports — and one thing it leaves unanswered.
No evidence pinned yet.
Work stays in this page session. Download before leaving. No pupil response is sent to a model or saved on a server.
What is real here?
The browser executes both blocks of Brightlab Binding Transformer v1: learned embeddings, causal attention, neural layers, normalisation and a next-word readout. The 10,522 parameters were trained in PyTorch. No answer is looked up from a prewritten result table.
- What it learned
- 230 fixed-format examples, four names and four colours. Its original result on 58 reserved combinations was 44 correct. We recompute that result in your browser when you run the test.
- What it cannot show
- It has a ten-word vocabulary. It cannot chat, reason about the world, or represent Goodfire’s much larger research models. Attention pictures and feature names are hypotheses to investigate.
- Research vocabulary
- Activation: a number produced for this input. Weight: a number learned during training. Checkpoint: a saved set of weights. Intervention: a deliberate change to a computation.
JavaScript computations checked against 576 native reference outputs, with maximum logit difference below 0.0001. Independent teaching adaptation; no Goodfire endorsement. Source review: 7 September 2026.
Downloaded model calculations work offline. External research links need internet access.