Year 12 · 55 minutes · Live trained model and learned-weight decomposition
Decompose the tiny transformer’s actual learned output matrix, edit a retained component, and rerun frozen evaluation prompts. Inspect both group performance and the weighted average.
Student paper journal · Interactive notebook · Editable Python
Try the starting controls, save two runs and reopen a downloaded project. Print the student journal before class. If Python cannot load, use the supplied example below for a prediction and evidence critique; rerunning changed inputs requires the live or native notebook. Do not present paper discussion as a new model experiment. Keep an individual record even when you investigate in pairs. Record the exact settings, source and data split.
Start with 5–8 minutes of prediction, use about half the lesson for paired comparisons, then reserve at least 10 minutes for a counterexample and an individual explanation. This is one learned-matrix experiment. Population reweighting changes the reported average, not the training data. SVD components are not established semantic mechanisms.
Settings: {'rank': 10, 'frequency': 0.09, 'edit': 0}
One experiment: decompose the trained output matrix, edit it, then rerun all 58 frozen evaluation prompts. Population-weighted loss: 1.4626.
| group | cases | base_loss | edited_loss | base_correct | edited_correct |
|---|---|---|---|---|---|
| Kiki queries | 12 | 3.367028881758401 | 3.367028881758399 | 8 | 8 |
| Other queries | 46 | 1.2742757112953083 | 1.2742757112953087 | 36 | 36 |
The table shows up to eight rows. Inspect the notebook for all values, controls and denominators.
Method and scope: W = UΣVᵀ. Keep r components, then add edit × σ₁u₁v₁ᵀ to the same learned readout matrix. Evaluate cross-entropy and exact answers with all other model weights fixed. The population slider reweights evaluation groups; it does not change training frequency. SVD is not SPD/VPD or K-FAC, and singular vectors are not verified semantic circuits.
Assess four criteria, each 0–2: testable prediction; comparison identifying what stayed fixed; accurate use of exact evidence; counterexample and limited conclusion. 0 means absent or contradicted by the record, 1 means partly supported, and 2 means clear and supported. A surprising result earns no penalty.
Beginning response: ‘The picture looks right.’ Ask for a particular case. Developing response: names one number without its setting. Ask what comparison supports it. Secure response: states the setting and measured change and separates the notebook's result from the source paper's claim. Extending response: authors or reserves a new case, tests an alternative explanation, and revises the claim if needed. These are marking examples, not pupil data.
Extension: Repeat on a different checkpoint and reserve new fact combinations before choosing an edit.
System and data: A four-layer 67M language model trained on the Pile
Method: Learns parameter subcomponents and causal importance with VPD; evaluates reconstruction, sparsity, interventions and robustness against alternative decompositions.
Task: Compare exact reconstruction, component removal and behavioural tests of a decomposition. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: Our SVD exercise is a contrast, not VPD. The paper studies a 67M-parameter model, not all frontier models.
System and data: Toy networks with known ground-truth mechanisms
Method: Learns parameter subcomponents using stochastic ablations and evaluates recovery of known computations; full-language-model scaling is a separate result.
Task: Distinguish reconstruction from causal component selection under ablations. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: SPD evidence here is based on controlled models; the SVD notebook is a baseline contrast, not an SPD implementation.
System and data: Language models and vision transformers with label noise
Method: Uses aggregate curvature approximations to identify and edit memorisation-associated weight directions, then checks collateral downstream effects.
Task: Compare a cheap direction under common-task curvature with its effect on a rare task. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: The source uses curvature approximations including K-FAC; our exact quadratic toy is not a K-FAC replication. Edits have collateral costs.
System and data: Companion to the 67M VPD language-model study
Method: Explains component reconstruction and learned causal importance; refers to the same experiment as Interpreting Language Model Parameters.
Task: Compare faithful reconstruction with simplicity and robustness of component explanations. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: This explains the same work as Interpreting Language Model Parameters; it is not a second independent result.
System and data: Power-law toy tasks and OLMo models from 4M to 4B
Method: Compares per-task losses, representations and gradient interference across capacity, including infrequent tasks; average loss can conceal retention failures.
Task: Inspect frequency-weighted loss and rare-task retention rather than only average performance. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: The reported mechanism is studied in particular toy and language models; size is not a guarantee on every task.
Independent Brightlab adaptations; no Goodfire endorsement or full reproduction claim. Source methods reviewed 7 September 2026. Curriculum connections are selected planning links; confirm your school syllabus.