brightlab.Made for curious mindsBrightlab
Year 12 · Evaluation
How can we tell which part of a system actually helped?
Go to the investigation ↓An ablation study with configuration matrix, paired traces and a defensible causal claim.
Take this investigation into Python: open the interactive Marimo notebook →
Which intervention caused the observed improvement?
An ablation removes or disables a system component while holding the evaluation suite fixed. This sandbox runs deterministic retrieval, filtering and review rules over structured synthetic cases. Paired results show which cases change when a component is removed. Components can interact: a filter may matter only when retrieval brings in hostile content. A score increase from an uncontrolled before/after comparison cannot identify a cause.
Before this lesson: Keep cases paired across treatments and interpret variation. Useful earlier investigations: The sealed test envelope; Confidence meets its evidence.
An ablation removes or disables a system component while holding the evaluation suite fixed. This sandbox runs deterministic retrieval, filtering and review rules over structured synthetic cases. Paired results show which cases change when a component is removed. Components can interact: a filter may matter only when retrieval brings in hostile content. A score increase from an uncontrolled before/after comparison cannot identify a cause.
Keep cases paired across treatments and interpret variation. Useful earlier investigations: The sealed test envelope; Confidence meets its evidence.
Plan three 50-minute sessions. Read the fixed synthetic suite including benign, missing-evidence and hostile-source cases. Explain the exact deterministic rules and keep expected outcomes sealed until predictions are recorded.
ACARA V9 AI curriculum connection ↗ · Technologies ↗
A complete teaching sequence · 150 minutes
Compare two fictional release reports that changed model, data and filter together.
Ask: “Which change caused the improvement?”
Listen for: “We cannot isolate it because several things changed.”
Predict which cases fail when retrieval alone is removed from the full system.
Ask: “Which cases depend on retrieved evidence?”
Listen for: “Those whose answer facts are only in a source.”
Toggle one component, keep fixtures fixed and inspect per-case transitions as well as aggregate pass counts. Save each configuration.
Ask: “Which exact test changed and why?”
Listen for: “Its required evidence or protection was removed.”
Compare filtering with retrieval on and off. Trace why filter benefit depends on hostile content reaching the system.
Ask: “Is the filter’s effect a single universal number?”
Listen for: “No, it depends on the rest of the pipeline.”
Run a small factorial comparison, declare hypotheses and document paired outcomes. Include a benign case where review imposes cost without changing correctness and a held-out challenge authored by peers.
Ask: “What remains uncertain after this fixed suite?”
Listen for: “Whether the cases represent real use and whether the effect transfers.”
Submit configuration, case-level differences, aggregate results and a narrow causal claim.
Ask: “What would invalidate your attribution?”
Listen for: “A changed dataset, seed, scoring rule or unrecorded component.”
A better score proves which feature helped.
The benefit of filtering depends on whether retrieval introduces hostile content; component effects are not simply additive.
Run a controlled factorial study and support one causal claim with matched case transitions and stated external-validity limits.
Require a record of fixed variables and one interaction explanation using case traces. Do not accept only bar-chart differences.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Experimental control | Compares changing systems and tests | Changes one component on a fixed suite | Uses factorial comparisons to expose interactions |
| Causal claim | Attributes any gain to the latest edit | Supports attribution with paired cases | Limits generalisation and tests a peer-authored challenge |
Start with two components and four cases; provide a configuration matrix before expanding the full suite.
Add repeated stochastic runs in the hardware pathway and report paired uncertainty intervals rather than one sample’s difference.
All cases and actions are synthetic dry runs. No real users are subjected to experimental treatments. Senior syllabus alignment requires local verification.
Shared device? Turn remembering off. A project file lets you continue on another device.
Your browser is the laboratory
Disable retrieval alone. Predict which fixed cases change before running the paired comparison.
Eight fixed cases include two source questions, two missing-evidence queries, two hostile sources, one ambiguous question and one simple answer. Retrieval, filtering and ambiguity review start enabled. Disable only one component first.
The experiment opens after your prediction.
There is no penalty for being surprised.
Apple Silicon · PyTorch MPS
Evaluate all eight retrieval/filter/review configurations over a repeated synthetic case suite on MPS/CUDA. Change case-family diversity and compare paired counts and component dependencies; repeats do not add independent evidence.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download Mac Studio investigation ↓unzip y12-ablation-mac-pathway.zip -d y12-ablation-mac cd y12-ablation-mac bash setup-mac.sh source .venv/bin/activate python experiment.py --device mps --output results marimo edit notebook.py
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
NVIDIA DGX Spark · PyTorch CUDA
Evaluate all eight retrieval/filter/review configurations over a repeated synthetic case suite on MPS/CUDA. Change case-family diversity and compare paired counts and component dependencies; repeats do not add independent evidence.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download DGX Spark investigation ↓unzip y12-ablation-dgx-pathway.zip -d y12-ablation-dgx cd y12-ablation-dgx bash run-dgx.sh # Open the localhost notebook URL printed by the container.
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
An ablation study with configuration matrix, paired traces and a defensible causal claim.
Download the editable handout →