Year 7 · 75 minutes · Evaluation
Which mistakes should an alert system try hardest to avoid?
A confusion matrix separates true and false alerts from true and false non-alerts. Precision asks how many alerts were correct; recall asks how many true targets were found. A threshold changes these counts. This lab adds explicit fictional costs for misses and false alarms. The lowest calculated cost depends on those assumptions and does not decide a real community’s values.
Print the fixed synthetic alert scores and reference labels. Calculate one confusion matrix manually. Set fictional miss and false-alarm costs, stressing that these are scenario inputs.
Count false positives and false negatives. Useful earlier investigations: y4-threshold Use pairs for investigation, with operator/reviewer swaps after each comparison. Keep individual predictions, journals and a short oral defence so group work does not hide understanding.
Australian Curriculum Version 9 · Digital Technologies: AC9TDI8P02, AC9TDI8P10. Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. ACARA AI curriculum connection · V9 Technologies These are planning connections, not ACARA endorsement or exhaustive descriptor alignment.
Place cards by prediction and reference truth.
Ask: “How is a false alarm different from a miss?”
Listen for: “One acts on a non-target; the other fails to act on a target.”
Predict alert count, precision and recall direction; allow uncertainty where the data may not be monotonic.
Ask: “Which cases will stop being alerts?”
Listen for: “Those between the old and new threshold.”
Adjust the threshold and inspect the matrix, fractions and cost equation. Record three settings with the same labelled cases.
Ask: “Why can accuracy hide a costly miss?”
Listen for: “It counts each correct case equally and ignores different costs.”
Set the threshold above all scores and inspect undefined precision rather than accepting a misleading perfect value.
Ask: “What is precision when there are no alerts?”
Listen for: “There is no denominator; it is undefined.”
Choose a threshold for supplied costs, then reverse the cost emphasis and reconsider. Document the people who would need to approve real costs.
Ask: “Did the scores change when our preferred threshold changed?”
Listen for: “No, our valuation of errors changed.”
Submit a matrix and cost calculation, including one zero-denominator case.
Ask: “Does minimum modelled cost prove a safe real system?”
Listen for: “No, the data and costs may omit important consequences.”
Accuracy is enough to choose a model.
Raise the threshold with scores and reference labels fixed. Predict alert and miss counts before seeing the matrix.
A no-alert policy can look accurate in a rare-target dataset while recall is zero and precision is undefined.
Choose and justify a threshold under two explicit cost scenarios; include an unresolved consequence.
Require manual calculation of one precision and recall fraction. Check that undefined metrics are not replaced with 0% or 100% without explanation.
Use ten cards and whole-number costs; keep the threshold to three candidate positions initially.
Compare a threshold chosen on validation data with a final sealed test set to expose selection bias.
An alert policy with confusion matrices, cost calculations and a decision-owner note.
The alert task is synthetic and low-stakes. Do not interpret the calculations as medical, emergency or safety guidance.
Broadcast 51 thresholds across a large synthetic labelled score population. Change the missed-case cost and compare exact true alerts, false alarms, misses and modelled cost.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Metric reasoning | Confuses precision and recall | Computes both from counts | Explains zero denominators and prevalence effects |
| Policy judgement | Optimises accuracy only | Uses explicit costs and error counts | Tests sensitivity to costs and names affected decision makers |
Australian Curriculum Version 9 · Digital Technologies
References: AC9TDI8P02, AC9TDI8P10. Read the current source (checked 2026-09-07).
Evidence to assess: An alert policy with confusion matrices, cost calculations and a decision-owner note.
Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. Moderate the supplied illustrative responses against your school unit and current achievement standard.
These are planning estimates to test with your class. A short session develops one supported claim; it does not compress the whole senior project.
| Stage | 45 minute focus | 60 minute investigation |
|---|---|---|
| Readiness and prediction | 0–5 | 0–5 |
| Trace the supplied example | 5–13 | 5–15 |
| Author and run cases | 13–25 | 15–35 |
| Counterexample and redesign | 25–35 | 35–45 |
| Explain and discuss | 35–42 | 45–55 |
| Export and handover | 42–45 | 55–60 |
For a longer project, use three 50-minute sessions. Session 1 (0–50): readiness, model, hypothesis and initial cases. Export a project and record the next test. Session 2 (50–100): reopen, check settings, author counterexamples and revise the design. Export the changed project and identify unresolved evidence. Session 3 (100–150): independent peer test, final artefact, individual explanation and moderation. If using two 60-minute sessions, stop at minute 60 after saving the first comparison; use 60–120 for redesign, independent test and defence.
Entry check: Count false positives and false negatives. Ask the learner to demonstrate it before choosing the level of support.
Preparation: allow about 15 minutes to run the starter, print the cards and check a project can be reopened. This estimate has not yet been measured in a classroom pilot.
Read the entry question aloud, model one row, and label the units. Offer the case table as a large-print sheet. Keep mathematical derivations optional until the learner can explain the comparison.
For one device, use a projector: one pair predicts, one operates, and the class records on paper. Swap roles after the first comparison. For individual access, support keyboard controls and a written table equivalent to each visual. Learners may explain orally or with an annotated diagram. Never require personal data, a recorded voice, or a photograph.
Mixed readiness: if the entry check is difficult, use the linked prerequisite and the first two case cards; retain the same central question. If secure, ask the learner to design an unseen test and state which explanation it could disprove.
Use the threshold to classify each case. Sum FP cost and FN cost separately, then compare alternatives.
Starting parameters: Boundary = 0.5, False alarm cost = 1, Miss cost = 5
TP 1; FP 1; FN 1; TN 1.
| case | score | actual | predicted | outcome | cost |
|---|---|---|---|---|---|
| 1 | 0.2 | False | False | TN | 0 |
| 2 | 0.4 | True | False | FN | 5 |
| 3 | 0.6 | False | True | FP | 1 |
| 4 | 0.8 | True | True | TP | 0 |
These are authored examples, not work collected from children. Assess reasoning using the lesson rubric, not whether the first prediction was correct.
Beginning: “It worked because the result looks right.” This identifies no exact case, control or measurement. Ask the learner to point to one row and say what happened.
Developing: “In the first case I recorded case: 1; score: 0.2; actual: False; predicted: False; outcome: TN; cost: 0.” This cites evidence, but does not yet explain how the result follows from the rule. Ask the learner to trace the relevant step.
Secure: “For the first supplied case, case: 1; score: 0.2; actual: False; predicted: False; outcome: TN; cost: 0. I can trace it using this mechanism: Use the threshold to classify each case. Sum FP cost and FN cost separately, then compare alternatives. My result supports a claim about these supplied cases. It does not establish that the same result holds outside them.” Look for an accurate trace, the actual settings and a bounded claim; accept equivalent oral or visual evidence.
Extending: The learner constructs and reruns a new case, reports whether the first explanation survives, and defends a revised design. Use this concrete challenge: Write a fictional scenario, assign costs before testing, and defend the threshold with per-case evidence. Require the original and changed evidence and this boundary: Costs are scenario assumptions chosen before evaluation, not objective values of people.
Moderation: first assess independently against each lesson criterion. Compare the exact trace or artefact that led to your judgement. Resolve differences using evidence, not polished language. Keep each learner's individual explanation even when the artefact was produced in a group.