brightlab.Made for curious mindsBrightlab
Year 9 · Uncertainty
When a model says 90%, how often should it be right?
Go to the investigation ↓A calibration report with reliability bins, Brier calculations and transfer limitations.
Take this investigation into Python: open the interactive Marimo notebook →
Does the stated probability match unseen outcomes?
Calibration compares predicted probabilities with observed frequencies across many cases. This binary classifier supplies logits and fixed outcomes. Dividing logits by a temperature changes confidence while preserving which side of zero they occupy, so threshold-0.5 accuracy can remain unchanged. Brier score is the mean squared probability error. Better calibration on these data is not a guarantee about an individual case or a shifted population.
Before this lesson: Interpret frequencies, probabilities and sample sizes. Useful earlier investigations: The cost of an alarm.
Calibration compares predicted probabilities with observed frequencies across many cases. This binary classifier supplies logits and fixed outcomes. Dividing logits by a temperature changes confidence while preserving which side of zero they occupy, so threshold-0.5 accuracy can remain unchanged. Brier score is the mean squared probability error. Better calibration on these data is not a guarantee about an individual case or a shifted population.
Interpret frequencies, probabilities and sample sizes. Useful earlier investigations: The cost of an alarm.
Prepare the fixed logit/outcome table and bin definitions. Explain that small bins are uncertain and empty bins should not display invented accuracy. Keep a second synthetic population for the edge case.
ACARA V9 AI curriculum connection ↗ · Technologies ↗
A complete teaching sequence · 80 minutes
Imagine ten comparable forecasts at 90% and discuss the expected frequency of positives.
Ask: “Does 90% mean this one case must be positive?”
Listen for: “No; it describes a frequency expectation across cases.”
Predict whether temperature changes the sign of logits or the sharpness of probabilities.
Ask: “Could accuracy stay fixed while probability quality changes?”
Listen for: “Yes, decisions can stay on the same side of 0.5.”
Adjust temperature and inspect bin counts, mean probabilities, observed frequencies and Brier score. Keep outcomes fixed.
Ask: “Which bin has enough cases to support a strong claim?”
Listen for: “We need its count, not just a neat point on the line.”
Evaluate the chosen temperature on a different supplied outcome distribution.
Ask: “Did calibration transfer automatically?”
Listen for: “No, the new population can differ.”
Choose temperature on the development set, freeze it and evaluate the alternate set. Report metrics and small-bin limitations.
Ask: “What should be monitored after deployment?”
Listen for: “Confidence versus outcomes on the relevant population.”
Submit one pair of changed probabilities with the same class prediction.
Ask: “Can this procedure fix a model that ranks examples badly?”
Listen for: “It mainly rescales confidence; it does not repair all discrimination errors.”
A score of 90% guarantees correctness.
Temperature can improve Brier score without changing any predicted class; calibration can fail again on a shifted population.
Choose a calibration setting on development data and report its frozen performance on the second population with bin counts.
Require a hand-calculated bin and one Brier contribution. Check that learners avoid interpreting a bin frequency as an individual guarantee.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Probability reasoning | Treats confidence as certainty | Compares confidence and frequency | Explains bin-size and individual-case limits |
| Calibration evaluation | Optimises the same score repeatedly | Separates development and evaluation | Reports transfer failure and monitoring needs |
Use probabilities in tenths and a small table of binary outcomes; provide squared-error examples.
Compare equal-width with equal-count bins and explain why the visual conclusion can change.
No individual risk estimates or real student outcomes are used. The probabilities belong only to synthetic fixtures.
Shared device? Turn remembering off. A project file lets you continue on another device.
Your browser is the laboratory
Increase temperature with logits fixed. Predict confidence concentration and whether 0.5-threshold labels change.
Forty logits range from −4 to 4 with supplied binary outcomes. Probability is sigmoid(logit / temperature). Classification uses a 0.5 cutoff; reliability is reported in five bins with counts.
The experiment opens after your prediction.
There is no penalty for being surprised.
Apple Silicon · PyTorch MPS
Evaluate temperature-scaled probabilities over a large seeded synthetic population. Compare reliability-bin counts, mean probabilities, observed frequencies and Brier error on MPS/CUDA.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download Mac Studio investigation ↓unzip y9-calibration-mac-pathway.zip -d y9-calibration-mac cd y9-calibration-mac bash setup-mac.sh source .venv/bin/activate python experiment.py --device mps --output results marimo edit notebook.py
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
NVIDIA DGX Spark · PyTorch CUDA
Evaluate temperature-scaled probabilities over a large seeded synthetic population. Compare reliability-bin counts, mean probabilities, observed frequencies and Brier error on MPS/CUDA.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download DGX Spark investigation ↓unzip y9-calibration-dgx-pathway.zip -d y9-calibration-dgx cd y9-calibration-dgx bash run-dgx.sh # Open the localhost notebook URL printed by the container.
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
A calibration report with reliability bins, Brier calculations and transfer limitations.
Download the editable handout →