Year 3 · 55 minutes · Evaluation
How do we find out whether a machine learned beyond its practice?
A lookup machine can remember exact examples. Repeating those examples tests its memory, not whether it handles a new input. We keep some examples out of training and use them once to test the finished system. This lab contrasts lookup with a simple size rule. Neither score should be trusted without knowing which examples were used and what the system could see.
Prepare twelve numbered synthetic object cards, with even and odd sizes distributed across the range. Put four in a sealed envelope before demonstrating lookup. Explain the fallback prediction for an unremembered value.
Separate examples used to choose a rule from examples used to test it. Useful earlier investigations: y2-coverage Use pairs with predictor/operator roles. Swap after the first comparison. Each learner draws or writes their own explanation using one exact case.
Australian Curriculum Version 9 · Digital Technologies: AC9TDI4P02, AC9TDI4P05. Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. ACARA AI curriculum connection · V9 Technologies These are planning connections, not ACARA endorsement or exhaustive descriptor alignment.
Read three card answers and ask a learner to repeat them. Introduce a new card.
Ask: “Did repeating the known cards prove the new answer?”
Listen for: “No; we had not tried that card.”
Choose a training partition and predict lookup performance on the sealed examples.
Ask: “Which cards can the lookup system remember?”
Listen for: “Only the training cards.”
Fit lookup and a size rule with the same partition. Reveal training and unseen scores, including denominators. Keep the test set fixed while comparing models.
Ask: “Which score answers our question about new examples?”
Listen for: “The unseen score.”
Move a previously tested item into training and re-evaluate it. Identify why its improved score is no longer independent.
Ask: “Is this still an unseen test?”
Listen for: “No; we taught the machine that answer.”
Reserve a fresh set on paper, state it before training and compare the two model types. Keep a record of every example used.
Ask: “What should another group know to trust your score?”
Listen for: “Which cards trained it and which tested it.”
Submit training count, holdout count, correct count and one error.
Ask: “What would four correct tests fail to prove?”
Listen for: “That every future example will be right.”
Remembering practice answers means learning.
Predict the memoriser’s unseen score before opening the holdout examples.
A lookup model scores perfectly on repeated practice examples but defaults incorrectly on unseen sizes.
Choose an independent partition, compare lookup with a size rule and explain any contamination caused by repeated tuning.
Check whether the student can identify a contaminated example. If not, physically move its card from the sealed envelope to the training pile.
Use six cards and count correct predictions with counters. Read score fractions aloud.
Create separate training, validation and final-test envelopes, using validation for one design change only.
A sealed-test protocol and a two-model comparison table.
All examples are synthetic object measurements. Do not turn classmates’ assessment results into training data.
Repeat train/holdout partitions with deterministic seeds to show score variation and the inflation caused by evaluating memorised examples.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Test independence | Reuses training examples as proof | Keeps holdout examples separate | Identifies contamination after tuning |
| Score reporting | Reports only a percentage | Includes counts and model type | Analyses an unseen error and limits the claim |
Australian Curriculum Version 9 · Digital Technologies
References: AC9TDI4P02, AC9TDI4P05. Read the current source (checked 2026-09-07).
Evidence to assess: A sealed-test protocol and a two-model comparison table.
Selected aspects only. This activity contributes evidence; it does not cover the full descriptor or achievement standard. A programming descriptor is not claimed for merely moving controls. Moderate the supplied illustrative responses against your school unit and current achievement standard.
These are planning estimates to test with your class. A short session develops one supported claim; it does not compress the whole senior project.
| Stage | 45 minute focus | 60 minute investigation |
|---|---|---|
| Readiness and prediction | 0–5 | 0–5 |
| Trace the supplied example | 5–13 | 5–15 |
| Author and run cases | 13–25 | 15–35 |
| Counterexample and redesign | 25–35 | 35–45 |
| Explain and discuss | 35–42 | 45–55 |
| Export and handover | 42–45 | 55–60 |
For a longer project, use three 50-minute sessions. Session 1 (0–50): readiness, model, hypothesis and initial cases. Export a project and record the next test. Session 2 (50–100): reopen, check settings, author counterexamples and revise the design. Export the changed project and identify unresolved evidence. Session 3 (100–150): independent peer test, final artefact, individual explanation and moderation. If using two 60-minute sessions, stop at minute 60 after saving the first comparison; use 60–120 for redesign, independent test and defence.
Entry check: Separate examples used to choose a rule from examples used to test it. Ask the learner to demonstrate it before choosing the level of support.
Preparation: allow about 15 minutes to run the starter, print the cards and check a project can be reopened. This estimate has not yet been measured in a classroom pilot.
Read the entry question aloud, model one row, and label the units. Offer the case table as a large-print sheet. Keep mathematical derivations optional until the learner can explain the comparison.
For one device, use a projector: one pair predicts, one operates, and the class records on paper. Swap roles after the first comparison. For individual access, support keyboard controls and a written table equivalent to each visual. Learners may explain orally or with an annotated diagram. Never require personal data, a recorded voice, or a photograph.
Mixed readiness: if the entry check is difficult, use the linked prerequisite and the first two case cards; retain the same central question. If secure, ask the learner to design an unseen test and state which explanation it could disprove.
Select a size rule on practice cards, then test it on sealed cards without changing the rule.
Starting parameters: Large from size = 5
Sealed cases revealed. Changing the rule now makes them development cases.
| size | colour | split | predicted | expected | correct |
|---|---|---|---|---|---|
| 2 | green | practice | small | small | True |
| 8 | orange | practice | large | large | True |
| 5 | green | sealed | large | large | True |
| 4 | orange | sealed | small | small | True |
These are authored examples, not work collected from children. Assess reasoning using the lesson rubric, not whether the first prediction was correct.
Beginning: “It worked because the result looks right.” This identifies no exact case, control or measurement. Ask the learner to point to one row and say what happened.
Developing: “In the first case I recorded size: 2; colour: green; split: practice; predicted: small; expected: small; correct: True.” This cites evidence, but does not yet explain how the result follows from the rule. Ask the learner to trace the relevant step.
Secure: “For the first supplied case, size: 2; colour: green; split: practice; predicted: small; expected: small; correct: True. I can trace it using this mechanism: Select a size rule on practice cards, then test it on sealed cards without changing the rule. My result supports a claim about these supplied cases. It does not establish that the same result holds outside them.” Look for an accurate trace, the actual settings and a bounded claim; accept equivalent oral or visual evidence.
Extending: The learner constructs and reruns a new case, reports whether the first explanation survives, and defends a revised design. Use this concrete challenge: Seal four new cards before selecting a rule. Record the rule before opening the envelope. Require the original and changed evidence and this boundary: Once sealed results influence your choice, those cards become development data.
Moderation: first assess independently against each lesson criterion. Compare the exact trace or artefact that led to your judgement. Resolve differences using evidence, not polished language. Keep each learner's individual explanation even when the artefact was produced in a group.