Year 11 · 120 minutes · Monitoring
How will we notice a deployed model is no longer behaving as expected?
Monitoring compares recent observations with a reference. This simulator advances a deterministic sensor stream with a known change point and optional short spike. A rolling mean can reveal sustained drift, but window length and alert threshold trade detection delay against false alarms. Detecting a changed input does not by itself show that prediction quality changed. A rollback needs a named owner and a known prior configuration.
Plan two sessions and print the synthetic stream with the teacher’s change-point key hidden initially. Distinguish input monitoring from delayed outcome evaluation. Rehearse reset and stepping to reproduce the same stream.
Compare a reference distribution with recent observations. Useful earlier investigations: y4-shift, y7-errors Use pairs for investigation, with operator/reviewer swaps after each comparison. Keep individual predictions, journals and a short oral defence so group work does not hide understanding.
Queensland Digital Solutions 2025 v1.4: Unit 3 · objective 3, Unit 3 · objective 4, Unit 3 · objective 7. Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. ACARA AI curriculum connection · V9 Technologies These are planning connections, not ACARA endorsement or exhaustive descriptor alignment.
State the reference mean and who would receive an alert in the fictional system.
Ask: “Is an alert useful if nobody knows what to do?”
Listen for: “No; it needs an owner and a response.”
Predict delay and spike sensitivity for short versus long windows.
Ask: “What can smoothing remove along with noise?”
Listen for: “It can delay evidence of a real change.”
Step observations, inspect the rolling window and alert condition, then replay with a different window from the same start.
Ask: “Which observations contribute to today’s mean?”
Listen for: “Only those inside the current window.”
Use a brief anomaly without sustained drift and count false alerts.
Ask: “Does one unusual value establish a new normal?”
Listen for: “No; it might be temporary.”
In session two, select window, threshold and required persistence. Test spike and shift streams, measure detection delay and define investigation, rollback and recovery steps.
Ask: “When would an input alert justify rollback?”
Listen for: “After the policy’s evidence and safety checks, not automatically from every spike.”
Another group advances the stream and follows the documented policy.
Ask: “What do you record after recovery?”
Listen for: “Trigger, evidence, action, owner and checks that normal operation returned.”
Deployment is the end of testing.
Increase the rolling window with the stream fixed. Predict effects on spike alerts and sustained-shift detection delay.
A short spike can trigger an input alert without a lasting distribution change; a long window can miss an important change until late.
Create and replay a monitoring/rollback policy with explicit false-alarm and detection-delay evidence.
Ask for the exact observations in one rolling window and the first valid alert time. Check that a human owner and recovery criterion are specified.
Use a ten-step stream and a three-point rolling average before exploring longer windows.
Compare mean-shift monitoring with an error-rate monitor using delayed labels; explain their different blind spots.
A tested monitoring runbook with alert trace, rollback owner and recovery checks.
The stream is synthetic and no production service is controlled. Avoid treating a statistical alarm as evidence about a particular person. Senior mapping remains jurisdiction-specific.
Simulate thousands of independently noisy streams with a fixed known change point. Change the rolling window and compare detection-delay and false-alert quantiles.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Monitoring mechanics | Reacts to every high point | Explains rolling mean and threshold | Quantifies false alarms and delay |
| Operational policy | Ends at an alert | Defines an owner and response | Tests rollback, recovery and limits of input-only evidence |
Queensland Digital Solutions 2025 v1.4
References: Unit 3 · objective 3, Unit 3 · objective 4, Unit 3 · objective 7. Read the current source (checked 2026-09-07).
Evidence to assess: Incident analysis, monitoring criterion and a justified refinement.
Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. A supporting classroom task, not a QCAA-approved assessment instrument. Teachers must set their own assessment conditions and confirm alignment with their course. Other jurisdictions require local mapping.
These are planning estimates to test with your class. A short session develops one supported claim; it does not compress the whole senior project.
| Stage | 45 minute focus | 60 minute investigation |
|---|---|---|
| Readiness and prediction | 0–5 | 0–5 |
| Trace the supplied example | 5–13 | 5–15 |
| Author and run cases | 13–25 | 15–35 |
| Counterexample and redesign | 25–35 | 35–45 |
| Explain and discuss | 35–42 | 45–55 |
| Export and handover | 42–45 | 55–60 |
For a longer project, use three 50-minute sessions. Session 1 (0–50): readiness, model, hypothesis and initial cases. Export a project and record the next test. Session 2 (50–100): reopen, check settings, author counterexamples and revise the design. Export the changed project and identify unresolved evidence. Session 3 (100–150): independent peer test, final artefact, individual explanation and moderation. If using two 60-minute sessions, stop at minute 60 after saving the first comparison; use 60–120 for redesign, independent test and defence.
Entry check: Compare a reference distribution with recent observations. Ask the learner to demonstrate it before choosing the level of support.
Preparation: allow about 25 minutes to run the starter, print the cards and check a project can be reopened. This estimate has not yet been measured in a classroom pilot.
Read the entry question aloud, model one row, and label the units. Offer the case table as a large-print sheet. Keep mathematical derivations optional until the learner can explain the comparison.
For one device, use a projector: one pair predicts, one operates, and the class records on paper. Swap roles after the first comparison. For individual access, support keyboard controls and a written table equivalent to each visual. Learners may explain orally or with an annotated diagram. Never require personal data, a recorded voice, or a photograph.
Mixed readiness: if the entry check is difficult, use the linked prerequisite and the first two case cards; retain the same central question. If secure, ask the learner to design an unseen test and state which explanation it could disprove.
Compare rolling means with the reference and separately count delayed error labels. An input alarm is not itself a performance failure.
Starting parameters: Reference mean = 10, Window = 3, Shift alarm = 3
Keep the input alarm separate from later observed errors; write the response decision in your report.
| step | value | window mean | alarm | later errors |
|---|---|---|---|---|
| 1 | 10 | 10 | False | 0 |
| 2 | 11 | 10.5 | False | 0 |
| 3 | 9 | 10 | False | 0 |
| 4 | 15 | 11.666667 | False | 0 |
| 5 | 16 | 13.333333 | True | 1 |
| 6 | 17 | 16 | True | 2 |
These are authored examples, not work collected from children. Assess reasoning using the lesson rubric, not whether the first prediction was correct.
Beginning: “It worked because the result looks right.” This identifies no exact case, control or measurement. Ask the learner to point to one row and say what happened.
Developing: “In the first case I recorded step: 1; value: 10; window mean: 10; alarm: False; later errors: 0.” This cites evidence, but does not yet explain how the result follows from the rule. Ask the learner to trace the relevant step.
Secure: “For the first supplied case, step: 1; value: 10; window mean: 10; alarm: False; later errors: 0. I can trace it using this mechanism: Compare rolling means with the reference and separately count delayed error labels. An input alarm is not itself a performance failure. My result supports a claim about these supplied cases. It does not establish that the same result holds outside them.” Look for an accurate trace, the actual settings and a bounded claim; accept equivalent oral or visual evidence.
Extending: The learner constructs and reruns a new case, reports whether the first explanation survives, and defends a revised design. Use this concrete challenge: Author an event stream, label delayed outcomes, and write an investigate/pause/release decision with an accountable role. Require the original and changed evidence and this boundary: Later labels are fictional and immediately supplied here; a real response plan must account for label delay.
Moderation: first assess independently against each lesson criterion. Compare the exact trace or artefact that led to your judgement. Resolve differences using evidence, not polished language. Keep each learner's individual explanation even when the artefact was produced in a group.