Year 6 · 35 minutes · Live trained model
Turn off one attention head in a tiny transformer and compare its answer with the original on exactly the same card.
Student paper journal · Interactive notebook · Editable Python
Try the starting controls, save two runs and reopen a downloaded project. Print the student journal before class. If Python cannot load, use the supplied example below for a prediction and evidence critique; rerunning changed inputs requires the live or native notebook. Do not present paper discussion as a new model experiment. Draw, point or explain aloud. A partner or adult may record your words.
Start with 5–8 minutes of prediction, use about half the lesson for paired comparisons, then reserve at least 10 minutes for a counterexample and an individual explanation. Explain that an intervention changes the computation. A small effect can mean redundancy or a poor test; it does not prove the part is useless.
Settings: {'case': 0, 'layer': 0, 'head': 0, 'authored': ''}
Prompt: “pip has red . wattle has blue . pip has …”. The written facts support red. Base model chooses red. Removing block 0, head 0 changes the answer to red.
| word | original_probability | probability |
|---|---|---|
| pip | 1.3e-05 | 9e-06 |
| wattle | 1e-05 | 9e-06 |
| kiki | 2.1e-05 | 2.1e-05 |
| bo | 8e-06 | 6e-06 |
| red | 0.999122 | 0.999906 |
| blue | 0.000798 | 5e-06 |
| green | 2e-06 | 7e-06 |
| gold | 2e-06 | 1.8e-05 |
The table shows up to eight rows. Inspect the notebook for all values, controls and denominators.
Method and scope: Ablation sets the chosen attention head output to zero before its output projection. It changes an internal computation, while weights and prompt stay fixed.
Assess four criteria, each 0–2: testable prediction; comparison identifying what stayed fixed; accurate use of exact evidence; counterexample and limited conclusion. 0 means absent or contradicted by the record, 1 means partly supported, and 2 means clear and supported. A surprising result earns no penalty.
Beginning response: ‘The picture looks right.’ Ask for a particular case. Developing response: names one number without its setting. Ask what comparison supports it. Secure response: states the setting and measured change and separates the notebook's result from the source paper's claim. Extending response: authors or reserves a new case, tests an alternative explanation, and revises the claim if needed. These are marking examples, not pupil data.
Extension: Compare both heads one at a time. Keep a log of probability changes even when the top answer stays the same.
System and data: Historical Llama steering demonstrations
Method: Changes selected internal feature strengths and inspects example outputs; demonstrations need independent task and side-effect checks.
Task: Turn one internal switch and look for side effects. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: Steering is not guaranteed control or a source of new knowledge. Ember links are historical.
System and data: Cross-domain interpretability perspective
Method: Builds an argument from existing examples and open problems; optimism is not a measured success rate.
Task: Write a hopeful idea and then a way it could be wrong. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: The author explicitly presents optimism amid unsolved scientific questions.
System and data: A four-layer 67M language model trained on the Pile
Method: Learns parameter subcomponents and causal importance with VPD; evaluates reconstruction, sparsity, interventions and robustness against alternative decompositions.
Task: Remove part of a working system and check what changes. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: Our SVD exercise is a contrast, not VPD. The paper studies a 67M-parameter model, not all frontier models.
System and data: Llama, Gemma and Qwen families, 2–72B, ten binding tasks
Method: Uses ablations to separate positional, lexical and reflexive retrieval; compares causal predictions as entity lists and context length change.
Task: Swap who owns each coloured object and see whether order matters. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: Our tiny binding transformer is independently trained and does not reproduce the paper's nine-model circuit findings.
System and data: Mechanistic interpretability survey
Method: Organises unresolved conceptual, methodological and practical questions; evaluate the evidence for each underlying method separately.
Task: What experiment would show that our explanation is wrong? Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: A research roadmap is not evidence that these problems have been solved.
System and data: Image-patch representations and a BatchTopK SAE
Method: Decomposes image representations into sparse features and edits selected patches; visual examples and reconstruction are distinct from reliable semantic control.
Task: Try changing one clue while watching for unwanted changes elsewhere. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: This is image generation, not an LLM experiment; the classroom intervention is a transferable analogy.
System and data: GPT-2 Small on a known greater-than task
Method: Trains cross-layer transcoders, generates attribution graphs and compares the recovered mechanism with a previously characterised circuit.
Task: A wiring picture needs a switch test to support its arrows. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: Replication found both similarities and differences. Our attention patching does not reproduce CLT circuit discovery.
System and data: Toy networks with known ground-truth mechanisms
Method: Learns parameter subcomponents using stochastic ablations and evaluates recovery of known computations; full-language-model scaling is a separate result.
Task: Compare removing one component with rebuilding the whole system. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: SPD evidence here is based on controlled models; the SVD notebook is a baseline contrast, not an SPD implementation.
System and data: Community synthesis of circuit methods and findings
Method: Compares probes, sparse decompositions, attribution graphs and causal testing across published work; synthesis is not a single replication.
Task: Explain why drawing arrows is different from testing a connection. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: The linked community guide was reviewed. Attribution graphs provide hypotheses with missing or uninterpretable components; interventions remain necessary.
System and data: Language models and vision transformers with label noise
Method: Uses aggregate curvature approximations to identify and edit memorisation-associated weight directions, then checks collateral downstream effects.
Task: Removing one learned detail may also damage another skill. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: The source uses curvature approximations including K-FAC; our exact quadratic toy is not a K-FAC replication. Edits have collateral costs.
System and data: Companion to the 67M VPD language-model study
Method: Explains component reconstruction and learned causal importance; refers to the same experiment as Interpreting Language Model Parameters.
Task: Ask whether the rebuilt parts still produce the original answer. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: This explains the same work as Interpreting Language Model Parameters; it is not a second independent result.
Independent Brightlab adaptations; no Goodfire endorsement or full reproduction claim. Source methods reviewed 7 September 2026. Curriculum connections are selected planning links; confirm your school syllabus.