Year 7 · 75 minutes · Evaluation
Is a brilliant score hiding information from the future?
A useful predictor may use only information available when the prediction is made. A post-event measurement can reveal the outcome and make evaluation look excellent. This lab compares a valid pre-event signal with a synthetic post-event indicator. Its high leaky score is real computation on unsuitable inputs. Removing leakage usually lowers apparent performance while making the evaluation more relevant.
Print the synthetic tide-event table with timestamps for forecast readings and post-event inspection. Decide the prediction deadline before showing scores. No actual tide forecast is used.
Separate training and testing and order events by time. Useful earlier investigations: y3-holdout Use pairs for investigation, with operator/reviewer swaps after each comparison. Keep individual predictions, journals and a short oral defence so group work does not hide understanding.
Australian Curriculum Version 9 · Digital Technologies: AC9TDI8P01, AC9TDI8P10. Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. ACARA AI curriculum connection · V9 Technologies These are planning connections, not ACARA endorsement or exhaustive descriptor alignment.
Place feature cards before or after the prediction time on a timeline.
Ask: “Could the system know the inspection result before the event?”
Listen for: “No, it is collected later.”
Predict which model scores higher before enabling the post-event feature.
Ask: “Why might the wrong feature make a score look better?”
Listen for: “It nearly contains the answer.”
Toggle the leaked feature while keeping events fixed. Inspect features, predictions and error counts. Record denominators.
Ask: “Is the calculation fake, or is the experiment invalid for the intended use?”
Listen for: “The calculation is real, but the input would not be available.”
Inspect the edge case in which the post-event feature has an innocent name such as final flag.
Ask: “Can a harmless column name make the feature valid?”
Listen for: “No; the timestamp and origin matter.”
Exclude late features, state a baseline and write a schema that records availability time. Re-evaluate and narrow the performance claim.
Ask: “Why is a lower honest score useful?”
Listen for: “It tells us what the system can really do at prediction time.”
A partner proposes a derived feature; decide whether its ingredients cross the boundary.
Ask: “Can a transformed late feature still leak?”
Listen for: “Yes, transforming it does not change when it became available.”
A very high test score always means a useful model.
Predict how adding a post-event indicator changes accuracy without changing the test events.
A renamed or transformed post-event flag still leaks future information even though its name looks harmless.
Create a feature-availability schema and evaluate a time-valid model against a declared baseline.
Ask learners to trace the origin of a derived field. Credit a lower score with a valid boundary over an unjustified perfect score.
Use a physical before/after timeline and four events before examining the full table.
Design a time-ordered split and explain why random row splitting can still leak repeated-event information.
A timestamped feature contract and a leaky/time-valid evaluation report.
Synthetic tide events only. These models are not forecasts and must not inform coastal safety decisions.
Evaluate thousands of synthetic events using a pre-event feature, a copied post-event outcome and a fixed baseline. Compare measured accuracy with whether the feature existed at prediction time.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Leak diagnosis | Trusts the largest score | Identifies a post-event input | Traces leakage through derived fields |
| Evaluation repair | Deletes columns without a rule | Enforces a stated prediction deadline | Compares a baseline and limits the resulting claim |
Australian Curriculum Version 9 · Digital Technologies
References: AC9TDI8P01, AC9TDI8P10. Read the current source (checked 2026-09-07).
Evidence to assess: A timestamped feature contract and a leaky/time-valid evaluation report.
Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. Moderate the supplied illustrative responses against your school unit and current achievement standard.
These are planning estimates to test with your class. A short session develops one supported claim; it does not compress the whole senior project.
| Stage | 45 minute focus | 60 minute investigation |
|---|---|---|
| Readiness and prediction | 0–5 | 0–5 |
| Trace the supplied example | 5–13 | 5–15 |
| Author and run cases | 13–25 | 15–35 |
| Counterexample and redesign | 25–35 | 35–45 |
| Explain and discuss | 35–42 | 45–55 |
| Export and handover | 42–45 | 55–60 |
For a longer project, use three 50-minute sessions. Session 1 (0–50): readiness, model, hypothesis and initial cases. Export a project and record the next test. Session 2 (50–100): reopen, check settings, author counterexamples and revise the design. Export the changed project and identify unresolved evidence. Session 3 (100–150): independent peer test, final artefact, individual explanation and moderation. If using two 60-minute sessions, stop at minute 60 after saving the first comparison; use 60–120 for redesign, independent test and defence.
Entry check: Separate training and testing and order events by time. Ask the learner to demonstrate it before choosing the level of support.
Preparation: allow about 15 minutes to run the starter, print the cards and check a project can be reopened. This estimate has not yet been measured in a classroom pilot.
Read the entry question aloud, model one row, and label the units. Offer the case table as a large-print sheet. Keep mathematical derivations optional until the learner can explain the comparison.
For one device, use a projector: one pair predicts, one operates, and the class records on paper. Swap roles after the first comparison. For individual access, support keyboard controls and a written table equivalent to each visual. Learners may explain orally or with an annotated diagram. Never require personal data, a recorded voice, or a photograph.
Mixed readiness: if the entry check is difficult, use the linked prerequisite and the first two case cards; retain the same central question. If secure, ask the learner to design an unseen test and state which explanation it could disprove.
A feature is eligible only if its availability time is no later than the prediction time.
Starting parameters: The supplied cases define the inputs.
3 cases calculated from your supplied inputs. Compare the evidence with your prediction.
| feature | available | prediction time | value | eligible |
|---|---|---|---|---|
| Morning attendance | 0 | 10 | 1 | True |
| Afternoon result | 60 | 10 | 1 | False |
| Yesterday result | -1440 | 10 | 0 | True |
These are authored examples, not work collected from children. Assess reasoning using the lesson rubric, not whether the first prediction was correct.
Beginning: “It worked because the result looks right.” This identifies no exact case, control or measurement. Ask the learner to point to one row and say what happened.
Developing: “In the first case I recorded feature: Morning attendance; available: 0; prediction time: 10; value: 1; eligible: True.” This cites evidence, but does not yet explain how the result follows from the rule. Ask the learner to trace the relevant step.
Secure: “For the first supplied case, feature: Morning attendance; available: 0; prediction time: 10; value: 1; eligible: True. I can trace it using this mechanism: A feature is eligible only if its availability time is no later than the prediction time. My result supports a claim about these supplied cases. It does not establish that the same result holds outside them.” Look for an accurate trace, the actual settings and a bounded claim; accept equivalent oral or visual evidence.
Extending: The learner constructs and reruns a new case, reports whether the first explanation survives, and defends a revised design. Use this concrete challenge: Author a timestamped feature table. Remove future information and explain why shuffled test rows did not fix leakage. Require the original and changed evidence and this boundary: Timestamp eligibility alone does not establish consent, usefulness or absence of other leakage.
Moderation: first assess independently against each lesson criterion. Compare the exact trace or artefact that led to your judgement. Resolve differences using evidence, not polished language. Keep each learner's individual explanation even when the artefact was produced in a group.