Year 9 · 80 minutes · Uncertainty
When a model says 90%, how often should it be right?
Calibration compares predicted probabilities with observed frequencies across many cases. This binary classifier supplies logits and fixed outcomes. Dividing logits by a temperature changes confidence while preserving which side of zero they occupy, so threshold-0.5 accuracy can remain unchanged. Brier score is the mean squared probability error. Better calibration on these data is not a guarantee about an individual case or a shifted population.
Prepare the fixed logit/outcome table and bin definitions. Explain that small bins are uncertain and empty bins should not display invented accuracy. Keep a second synthetic population for the edge case.
Interpret frequencies, probabilities and sample sizes. Useful earlier investigations: y7-errors Use pairs for investigation, with operator/reviewer swaps after each comparison. Keep individual predictions, journals and a short oral defence so group work does not hide understanding.
Australian Curriculum Version 9 · Digital Technologies: AC9TDI10P02, AC9TDI10P06. Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. ACARA AI curriculum connection · V9 Technologies These are planning connections, not ACARA endorsement or exhaustive descriptor alignment.
Imagine ten comparable forecasts at 90% and discuss the expected frequency of positives.
Ask: “Does 90% mean this one case must be positive?”
Listen for: “No; it describes a frequency expectation across cases.”
Predict whether temperature changes the sign of logits or the sharpness of probabilities.
Ask: “Could accuracy stay fixed while probability quality changes?”
Listen for: “Yes, decisions can stay on the same side of 0.5.”
Adjust temperature and inspect bin counts, mean probabilities, observed frequencies and Brier score. Keep outcomes fixed.
Ask: “Which bin has enough cases to support a strong claim?”
Listen for: “We need its count, not just a neat point on the line.”
Evaluate the chosen temperature on a different supplied outcome distribution.
Ask: “Did calibration transfer automatically?”
Listen for: “No, the new population can differ.”
Choose temperature on the development set, freeze it and evaluate the alternate set. Report metrics and small-bin limitations.
Ask: “What should be monitored after deployment?”
Listen for: “Confidence versus outcomes on the relevant population.”
Submit one pair of changed probabilities with the same class prediction.
Ask: “Can this procedure fix a model that ranks examples badly?”
Listen for: “It mainly rescales confidence; it does not repair all discrimination errors.”
A score of 90% guarantees correctness.
Increase temperature with logits fixed. Predict confidence concentration and whether 0.5-threshold labels change.
Temperature can improve Brier score without changing any predicted class; calibration can fail again on a shifted population.
Choose a calibration setting on development data and report its frozen performance on the second population with bin counts.
Require a hand-calculated bin and one Brier contribution. Check that learners avoid interpreting a bin frequency as an individual guarantee.
Use probabilities in tenths and a small table of binary outcomes; provide squared-error examples.
Compare equal-width with equal-count bins and explain why the visual conclusion can change.
A calibration report with reliability bins, Brier calculations and transfer limitations.
No individual risk estimates or real student outcomes are used. The probabilities belong only to synthetic fixtures.
Evaluate temperature-scaled probabilities over a large seeded synthetic population. Compare reliability-bin counts, mean probabilities, observed frequencies and Brier error on MPS/CUDA.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Probability reasoning | Treats confidence as certainty | Compares confidence and frequency | Explains bin-size and individual-case limits |
| Calibration evaluation | Optimises the same score repeatedly | Separates development and evaluation | Reports transfer failure and monitoring needs |
Australian Curriculum Version 9 · Digital Technologies
References: AC9TDI10P02, AC9TDI10P06. Read the current source (checked 2026-09-07).
Evidence to assess: A calibration report with reliability bins, Brier calculations and transfer limitations.
Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. Moderate the supplied illustrative responses against your school unit and current achievement standard.
These are planning estimates to test with your class. A short session develops one supported claim; it does not compress the whole senior project.
| Stage | 45 minute focus | 60 minute investigation |
|---|---|---|
| Readiness and prediction | 0–5 | 0–5 |
| Trace the supplied example | 5–13 | 5–15 |
| Author and run cases | 13–25 | 15–35 |
| Counterexample and redesign | 25–35 | 35–45 |
| Explain and discuss | 35–42 | 45–55 |
| Export and handover | 42–45 | 55–60 |
For a longer project, use three 50-minute sessions. Session 1 (0–50): readiness, model, hypothesis and initial cases. Export a project and record the next test. Session 2 (50–100): reopen, check settings, author counterexamples and revise the design. Export the changed project and identify unresolved evidence. Session 3 (100–150): independent peer test, final artefact, individual explanation and moderation. If using two 60-minute sessions, stop at minute 60 after saving the first comparison; use 60–120 for redesign, independent test and defence.
Entry check: Interpret frequencies, probabilities and sample sizes. Ask the learner to demonstrate it before choosing the level of support.
Preparation: allow about 15 minutes to run the starter, print the cards and check a project can be reopened. This estimate has not yet been measured in a classroom pilot.
Read the entry question aloud, model one row, and label the units. Offer the case table as a large-print sheet. Keep mathematical derivations optional until the learner can explain the comparison.
For one device, use a projector: one pair predicts, one operates, and the class records on paper. Swap roles after the first comparison. For individual access, support keyboard controls and a written table equivalent to each visual. Learners may explain orally or with an annotated diagram. Never require personal data, a recorded voice, or a photograph.
Mixed readiness: if the entry check is difficult, use the linked prerequisite and the first two case cards; retain the same central question. If secure, ask the learner to design an unseen test and state which explanation it could disprove.
Transform each logit by 1/temperature and report per-case squared probability error, grouped by split.
Starting parameters: Temperature = 1
fit Brier 0.34; validation Brier 0.16; final Brier 0.25
| original | calibrated | outcome | split | squared error |
|---|---|---|---|---|
| 0.8 | 0.8 | 1 | fit | 0.04 |
| 0.8 | 0.8 | 0 | fit | 0.64 |
| 0.6 | 0.6 | 1 | validation | 0.16 |
| 0.4 | 0.4 | 0 | validation | 0.16 |
| 0.9 | 0.9 | 1 | final | 0.01 |
| 0.7 | 0.7 | 0 | final | 0.49 |
These are authored examples, not work collected from children. Assess reasoning using the lesson rubric, not whether the first prediction was correct.
Beginning: “It worked because the result looks right.” This identifies no exact case, control or measurement. Ask the learner to point to one row and say what happened.
Developing: “In the first case I recorded original: 0.8; calibrated: 0.8; outcome: 1; split: fit; squared error: 0.04.” This cites evidence, but does not yet explain how the result follows from the rule. Ask the learner to trace the relevant step.
Secure: “For the first supplied case, original: 0.8; calibrated: 0.8; outcome: 1; split: fit; squared error: 0.04. I can trace it using this mechanism: Transform each logit by 1/temperature and report per-case squared probability error, grouped by split. My result supports a claim about these supplied cases. It does not establish that the same result holds outside them.” Look for an accurate trace, the actual settings and a bounded claim; accept equivalent oral or visual evidence.
Extending: The learner constructs and reruns a new case, reports whether the first explanation survives, and defends a revised design. Use this concrete challenge: Fit a calibration parameter on one split, choose it using validation, then reveal the untouched final split. Require the original and changed evidence and this boundary: A few cases cannot establish population calibration. Final rows must remain hidden until the choice is locked.
Moderation: first assess independently against each lesson criterion. Compare the exact trace or artefact that led to your judgement. Resolve differences using evidence, not polished language. Keep each learner's individual explanation even when the artefact was produced in a group.