0110 min
Sort the four outcomes
Place cards by prediction and reference truth.
Ask: “How is a false alarm different from a miss?”
Listen for: “One acts on a non-target; the other fails to act on a target.”
Brightlab
brightlab.Made for curious mindsbrightlab.Made for curious mindsBrightlab
Year 7 · Evaluation
Which mistakes should an alert system try hardest to avoid?
Go to the investigation ↓An alert policy with confusion matrices, cost calculations and a decision-owner note.
Which alarm mistakes matter in this situation?
A confusion matrix separates true and false alerts from true and false non-alerts. Precision asks how many alerts were correct; recall asks how many true targets were found. A threshold changes these counts. This lab adds explicit fictional costs for misses and false alarms. The lowest calculated cost depends on those assumptions and does not decide a real community’s values.
Before this lesson: Count false positives and false negatives. Useful earlier investigations: The uncertain recycling gate.
A confusion matrix separates true and false alerts from true and false non-alerts. Precision asks how many alerts were correct; recall asks how many true targets were found. A threshold changes these counts. This lab adds explicit fictional costs for misses and false alarms. The lowest calculated cost depends on those assumptions and does not decide a real community’s values.
Count false positives and false negatives. Useful earlier investigations: The uncertain recycling gate.
Print the fixed synthetic alert scores and reference labels. Calculate one confusion matrix manually. Set fictional miss and false-alarm costs, stressing that these are scenario inputs.
ACARA V9 AI curriculum connection ↗ · Technologies ↗
A complete teaching sequence · 75 minutes
Place cards by prediction and reference truth.
Ask: “How is a false alarm different from a miss?”
Listen for: “One acts on a non-target; the other fails to act on a target.”
Predict alert count, precision and recall direction; allow uncertainty where the data may not be monotonic.
Ask: “Which cases will stop being alerts?”
Listen for: “Those between the old and new threshold.”
Adjust the threshold and inspect the matrix, fractions and cost equation. Record three settings with the same labelled cases.
Ask: “Why can accuracy hide a costly miss?”
Listen for: “It counts each correct case equally and ignores different costs.”
Set the threshold above all scores and inspect undefined precision rather than accepting a misleading perfect value.
Ask: “What is precision when there are no alerts?”
Listen for: “There is no denominator; it is undefined.”
Choose a threshold for supplied costs, then reverse the cost emphasis and reconsider. Document the people who would need to approve real costs.
Ask: “Did the scores change when our preferred threshold changed?”
Listen for: “No, our valuation of errors changed.”
Submit a matrix and cost calculation, including one zero-denominator case.
Ask: “Does minimum modelled cost prove a safe real system?”
Listen for: “No, the data and costs may omit important consequences.”
Accuracy is enough to choose a model.
A no-alert policy can look accurate in a rare-target dataset while recall is zero and precision is undefined.
Choose and justify a threshold under two explicit cost scenarios; include an unresolved consequence.
Require manual calculation of one precision and recall fraction. Check that undefined metrics are not replaced with 0% or 100% without explanation.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Metric reasoning | Confuses precision and recall | Computes both from counts | Explains zero denominators and prevalence effects |
| Policy judgement | Optimises accuracy only | Uses explicit costs and error counts | Tests sensitivity to costs and names affected decision makers |
Use ten cards and whole-number costs; keep the threshold to three candidate positions initially.
Compare a threshold chosen on validation data with a final sealed test set to expose selection bias.
The alert task is synthetic and low-stakes. Do not interpret the calculations as medical, emergency or safety guidance.
Shared device? Turn remembering off. A project file lets you continue on another device.
Your browser is the laboratory
Raise the threshold with scores and reference labels fixed. Predict alert and miss counts before seeing the matrix.
Twenty scores are 1/21 through 20/21. Targets are cases 6,10,14,17,19,20. Alert threshold starts at 0.5; a miss costs eight fictional units and a false alarm costs one.
The experiment opens after your prediction.
There is no penalty for being surprised.
Apple Silicon · PyTorch MPS
Broadcast 51 thresholds across a large synthetic labelled score population. Change the missed-case cost and compare exact true alerts, false alarms, misses and modelled cost.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download Mac Studio investigation ↓unzip y7-errors-mac-pathway.zip -d y7-errors-mac cd y7-errors-mac bash setup-mac.sh source .venv/bin/activate python experiment.py --device mps --output results marimo edit notebook.py
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
NVIDIA DGX Spark · PyTorch CUDA
Broadcast 51 thresholds across a large synthetic labelled score population. Change the missed-case cost and compare exact true alerts, false alarms, misses and modelled cost.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download DGX Spark investigation ↓unzip y7-errors-dgx-pathway.zip -d y7-errors-dgx cd y7-errors-dgx bash run-dgx.sh # Open the localhost notebook URL printed by the container.
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
An alert policy with confusion matrices, cost calculations and a decision-owner note.
Download the editable handout →