Year 12 · 55 minutes · Two live trained checkpoints
Compare two actual tiny-transformer checkpoints. One was deliberately taught a biased Kiki colour rule; amplify their logit differences and inspect held-out effects.
Student paper journal · Interactive notebook · Editable Python
Try the starting controls, save two runs and reopen a downloaded project. Print the student journal before class. If Python cannot load, use the supplied example below for a prediction and evidence critique; rerunning changed inputs requires the live or native notebook. Do not present paper discussion as a new model experiment. Keep an individual record even when you investigate in pairs. Record the exact settings, source and data split.
Start with 5–8 minutes of prediction, use about half the lesson for paired comparisons, then reserve at least 10 minutes for a counterexample and an individual explanation. Neither checkpoint is a frontier LLM. The base model already makes errors; compare paired cases rather than attributing every error to fine-tuning.
Settings: {'case': 0, 'alpha': 1, 'authored': ''}
Prompt: “pip has red . wattle has blue . pip has …”. The written facts support red. Base model chooses red. At amplification 1, the answer is red. Check Kiki and non-Kiki cases.
| word | base | edited | probability |
|---|---|---|---|
| pip | 1.3148161001662482e-05 | 7.713648064733414e-06 | 7.713648064733414e-06 |
| wattle | 1.045875097946677e-05 | 7.3244410650241065e-06 | 7.3244410650241065e-06 |
| kiki | 2.0861183920034614e-05 | 1.571593013369778e-05 | 1.571593013369778e-05 |
| bo | 7.944501396839062e-06 | 5.894900355907713e-06 | 5.894900355907713e-06 |
| red | 0.9991223587857547 | 0.9989212477456781 | 0.9989212477456781 |
| blue | 0.0007984807143590375 | 0.001000181507956996 | 0.001000181507956996 |
| green | 2.114718686843653e-06 | 2.4522591609190436e-05 | 2.4522591609190436e-05 |
| gold | 2.3764371638909063e-06 | 1.5515904378333983e-06 | 1.5515904378333983e-06 |
The table shows up to eight rows. Inspect the notebook for all values, controls and denominators.
Method and scope: Amplified logits = base + α × (edited − base). α=0 is base; α=1 is the edited checkpoint. Fine-tuning deliberately assigned green to every Kiki query. Discovery under amplification is not natural failure prevalence.
Assess four criteria, each 0–2: testable prediction; comparison identifying what stayed fixed; accurate use of exact evidence; counterexample and limited conclusion. 0 means absent or contradicted by the record, 1 means partly supported, and 2 means clear and supported. A surprising result earns no penalty.
Beginning response: ‘The picture looks right.’ Ask for a particular case. Developing response: names one number without its setting. Ask what comparison supports it. Secure response: states the setting and measured change and separates the notebook's result from the source paper's claim. Extending response: authors or reserves a new case, tests an alternative explanation, and revises the claim if needed. These are marking examples, not pupil data.
Extension: Reproduce training, alter the fraction of biased labels, and reserve a fresh final test suite.
System and data: Research agenda with worked training examples
Method: Proposes observing per-example learning signals and changing what training generalises; separate aspirations from demonstrated interventions.
Task: Distinguish editing an activation from changing the training signal and checking regressions. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: This is a research agenda; intentional design is not a solved capability.
System and data: Paired language-model checkpoints and controlled fine-tunes
Method: Amplifies output-score differences to expose changed behaviour; increased discovery frequency is not natural failure prevalence.
Task: Calculate base plus alpha times edited-minus-base logits, then inspect a harmless learned colour bias. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: Amplified discovery is not an estimate of how often the original model fails.
System and data: Llama base models with Dolci and Tulu preference data
Method: Inspects predicted per-example learning effects before training, then compares intended and unintended signals in realistic preference datasets.
Task: Compare a base checkpoint with deliberate biased fine-tuning and test collateral changes. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: Our supervised colour task illustrates data effects; it is not a replication of contrastive-SAE post-training or DPO.
System and data: OLMo 2 7B SFT/DPO, preference data and 120 held-out LMSYS prompts
Method: Matches behaviour-difference activation vectors to preference-pair vectors, then filters or swaps ranked data and retrains to test the attribution causally.
Task: Distinguish a ranking signal from causal evidence obtained by retraining on edited data. Record the source model/task, a baseline, the changed factor, a measurement and an alternative explanation. State exactly which part your notebook investigates and which part it does not reproduce.
Boundary: The paper evaluates particular models and behaviours. Our checkpoint comparison does not implement its attribution algorithm.
Independent Brightlab adaptations; no Goodfire endorsement or full reproduction claim. Source methods reviewed 7 September 2026. Curriculum connections are selected planning links; confirm your school syllabus.