Year 6 · 35 minutes · Synthetic evaluation
Test a fictional checking rule on 20 labelled cases. Move its threshold and try a set where the clue stops working.
Student paper journal · Interactive notebook · Editable Python
Try the starting controls, save two runs and reopen a downloaded project. Print the student journal before class. If Python cannot load, use the supplied example below for a prediction and evidence critique; rerunning changed inputs requires the live or native notebook. Do not present paper discussion as a new model experiment. Draw, point or explain aloud. A partner or adult may record your words.
Start with 5–8 minutes of prediction, use about half the lesson for paired comparisons, then reserve at least 10 minutes for a counterexample and an individual explanation. Keep both types of error visible. Scores are invented fixtures, not probabilities from an actual model. Use only fictional cases, never classmates’ personal information.
Settings: {'threshold': 0.5, 'shift': 0}
Out of 10 cases needing a check, the rule catches 8 and misses 2. Among 10 other cases, it raises 4 false alarms.
| case | needs_check | clue_score | flagged | outcome |
|---|---|---|---|---|
| 1 | True | 0.45 | False | missed |
| 2 | False | 0.238 | False | correct pass |
| 3 | True | 0.675 | True | caught |
| 4 | False | 0.513 | True | false alarm |
| 5 | True | 0.9 | True | caught |
| 6 | False | 0.1 | False | correct pass |
| 7 | True | 0.562 | True | caught |
| 8 | False | 0.375 | False | correct pass |
The table shows up to eight rows. Inspect the notebook for all values, controls and denominators.
Method and scope: Flag a case when its hand-made clue score ≥ threshold. The shifted set reverses the association. These are fictional test fixtures, not measured LLM scores.
Assess four criteria, each 0–2: testable prediction; comparison identifying what stayed fixed; accurate use of exact evidence; counterexample and limited conclusion. 0 means absent or contradicted by the record, 1 means partly supported, and 2 means clear and supported. A surprising result earns no penalty.
Beginning response: ‘The picture looks right.’ Ask for a particular case. Developing response: names one number without its setting. Ask what comparison supports it. Secure response: states the setting and measured change and separates the notebook's result from the source paper's claim. Extending response: authors or reserves a new case, tests an alternative explanation, and revises the claim if needed. These are marking examples, not pupil data.
Extension: Design two new cases that challenge the rule and specify their correct labels before testing.
System and data: Engineering perspective; examples of safety infrastructure
Method: Reason through deployment, testing and operational failure modes; no new controlled model experiment.
Task: Keep a test record another class can repeat. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: An engineering argument, not experimental proof of model safety.
System and data: Research funding programme
Method: Describes eligibility and support; assess proposed experiments separately from the announcement.
Task: Choose a question that a small classroom experiment can answer. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: A grants announcement supplies opportunities, not a new scientific result.
System and data: Research fellowship programme
Method: Describes research directions and participation; does not test a scientific hypothesis.
Task: Take turns as predictor, tester and evidence checker. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: A fellowship announcement is careers context, not a research finding.
System and data: Genomic research collaboration
Method: Describes planned interpretation of biological model representations; requires later independent scientific validation.
Task: Ask who could check a scientific model's guess. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: A collaboration announcement is not clinical validation. No pupil health data are used.
System and data: Policies for interpretability tools
Method: Describes moderation, feature access and research processes; organisational policy and model capability are separate claims.
Task: Test a rule on both easy cases and awkward counterexamples. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: A stated safety process is not proof that all failure modes are covered.
System and data: Materials-discovery partnership
Method: Announces collaboration; a predicted material must still be independently evaluated.
Task: A promising suggestion still needs a real-world test. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: The announcement is not a materials experiment or an LLM result.
System and data: Organisational security-control audit
Method: Reports independent control assurance over an audit period; does not evaluate the truth of generated answers.
Task: Sort a security claim from an answer-correctness claim. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: SOC 2 is not a certificate of LLM truthfulness or absence of harmful behaviour.
System and data: Scribe experiments on Evo 2 and GPT-2 examples
Method: Reports agent-assisted research tasks and failure cases; provenance checks are needed when agents can shortcut work.
Task: Check whether a helper used the promised data and rules. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: The shared task suite is described as directional, not a rigorously audited benchmark.
System and data: Evo 2 embeddings and labelled genetic variants
Method: Fits variant-effect predictors and constructs evidence-based mechanistic hypotheses; abstract and research report were available for review.
Task: Separate a model's prediction from a scientist's confirmed explanation. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: Genomic predictions are hypotheses, not clinical conclusions. The linked preprint abstract and Goodfire report were reviewed; full preprint text was unavailable.
System and data: Pleiades and cell-free DNA pilot cohorts
Method: Uses model interpretation to propose fragment-length signals, then tests a simpler classifier on an independent cohort with uncertainty intervals.
Task: An interesting clue needs a separate checking set. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: This is a small-cohort biomedical study, not a diagnostic classroom tool or an LLM result.
System and data: Kimi K2.5 and Qwen3-32B; Fortress/Petri evaluation settings compared with WildChat conversations
Method: At sentence boundaries, scores a prespecified evaluation-awareness phrase from token probabilities and compares the monitor with explicit verbalisation and judge-based monitoring. Reported rollout savings are specific to these experiments; the score is a proxy for awareness, not a direct mental-state measurement.
Task: A clue can raise suspicion without proving why a character acted. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: Phrase choice and prompt framing matter. High AUROC does not establish intent or universal reliability.
System and data: Qwen3-4B and an abliterated variant on HarmBench, with a 100,000-rollout reference
Method: Interpolates the paired models in logit space, measures compliance along the path, fits an empirical log-linear trend below 50% compliance and extrapolates to the original model. Requires a related variant and a suitable trend; the classroom Wilson interval uses a different, direct-sampling method.
Task: Seeing no mistakes in a small sample does not prove there are none. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: The reported efficiency is setting-dependent. Our binomial experiment explains uncertainty; it does not implement the paper's extrapolation method.
System and data: Llama 3.1 8B sidecar, English/Japanese synthetic training and real production tests
Method: Compares activation, attention and frozen-SAE probes for token-level PII detection across synthetic-to-real shift, label noise and language settings.
Task: A clue detector needs both matching examples and tricky non-matches. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: Results depend on data and baselines; SAEs are not always superior. Classroom fixtures contain no real personal information.
System and data: Gemma-3-12B-IT and LongFact++ with 999 held-out prompts
Method: Trains factuality/correction probes and uses a frozen model for rewards; compares RL, inline intervention and best-of-N contributions with independent labels.
Task: A reward can encourage a shortcut if it checks the wrong thing. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: The paper combines methods; its headline reduction is not attributable to feature rewards alone. Our reward pool is synthetic.
System and data: Band-gap-conditioned MatterGen diffusion
Method: Uses an activation probe to accept or reject proposed denoising steps and evaluates targeting, stability, uniqueness and novelty.
Task: The best-scoring suggestion still needs an independent check. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: A model's predicted material property is not a physical measurement; this is not text-LLM research.
System and data: Eight models, nineteen benchmarks; causal work on Kimi K2.5/Fortress
Method: Manually verifies verbalised awareness, compares matched cues and performs interventions; broad correlations and narrower causal evidence have different scope.
Task: Keep the task the same and change only the 'this is a test' cue. Use the linked notebook to make two observations. Draw or describe one result and one thing this activity cannot tell us about the source system.
Boundary: Correlations span more models than the causal intervention study. Silence about a test does not prove absence of awareness.
Independent Brightlab adaptations; no Goodfire endorsement or full reproduction claim. Source methods reviewed 7 September 2026. Curriculum connections are selected planning links; confirm your school syllabus.