brightlab.Made for curious mindsBrightlab
Year 3 · Evaluation
How do we find out whether a machine learned beyond its practice?
Go to the investigation ↓A sealed-test protocol and a two-model comparison table.
Can a rule pass the cards it practised and fail new cards?
A lookup machine can remember exact examples. Repeating those examples tests its memory, not whether it handles a new input. We keep some examples out of training and use them once to test the finished system. This lab contrasts lookup with a simple size rule. Neither score should be trusted without knowing which examples were used and what the system could see.
Before this lesson: Separate examples used to choose a rule from examples used to test it. Useful earlier investigations: Beyond the collecting window.
A lookup machine can remember exact examples. Repeating those examples tests its memory, not whether it handles a new input. We keep some examples out of training and use them once to test the finished system. This lab contrasts lookup with a simple size rule. Neither score should be trusted without knowing which examples were used and what the system could see.
Separate examples used to choose a rule from examples used to test it. Useful earlier investigations: Beyond the collecting window.
Prepare twelve numbered synthetic object cards, with even and odd sizes distributed across the range. Put four in a sealed envelope before demonstrating lookup. Explain the fallback prediction for an unremembered value.
ACARA V9 AI curriculum connection ↗ · Technologies ↗
A complete teaching sequence · 55 minutes
Read three card answers and ask a learner to repeat them. Introduce a new card.
Ask: “Did repeating the known cards prove the new answer?”
Listen for: “No; we had not tried that card.”
Choose a training partition and predict lookup performance on the sealed examples.
Ask: “Which cards can the lookup system remember?”
Listen for: “Only the training cards.”
Fit lookup and a size rule with the same partition. Reveal training and unseen scores, including denominators. Keep the test set fixed while comparing models.
Ask: “Which score answers our question about new examples?”
Listen for: “The unseen score.”
Move a previously tested item into training and re-evaluate it. Identify why its improved score is no longer independent.
Ask: “Is this still an unseen test?”
Listen for: “No; we taught the machine that answer.”
Reserve a fresh set on paper, state it before training and compare the two model types. Keep a record of every example used.
Ask: “What should another group know to trust your score?”
Listen for: “Which cards trained it and which tested it.”
Submit training count, holdout count, correct count and one error.
Ask: “What would four correct tests fail to prove?”
Listen for: “That every future example will be right.”
Remembering practice answers means learning.
A lookup model scores perfectly on repeated practice examples but defaults incorrectly on unseen sizes.
Choose an independent partition, compare lookup with a size rule and explain any contamination caused by repeated tuning.
Check whether the student can identify a contaminated example. If not, physically move its card from the sealed envelope to the training pile.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Test independence | Reuses training examples as proof | Keeps holdout examples separate | Identifies contamination after tuning |
| Score reporting | Reports only a percentage | Includes counts and model type | Analyses an unseen error and limits the claim |
Use six cards and count correct predictions with counters. Read score fractions aloud.
Create separate training, validation and final-test envelopes, using validation for one design change only.
All examples are synthetic object measurements. Do not turn classmates’ assessment results into training data.
Shared device? Turn remembering off. A project file lets you continue on another device.
Your browser is the laboratory
Predict the memoriser’s unseen score before opening the holdout examples.
Twelve sizes arrive in order: 2,8,4,10,1,7,3,9,5,11,6,12. Sizes 7 or greater have label 1. The first six train the model. Lookup returns 0 for an unseen size; the alternative learns a size threshold.
The experiment opens after your prediction.
There is no penalty for being surprised.
Apple Silicon · PyTorch MPS
Repeat train/holdout partitions with deterministic seeds to show score variation and the inflation caused by evaluating memorised examples.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download Mac Studio investigation ↓unzip y3-holdout-mac-pathway.zip -d y3-holdout-mac cd y3-holdout-mac bash setup-mac.sh source .venv/bin/activate python experiment.py --device mps --output results marimo edit notebook.py
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
NVIDIA DGX Spark · PyTorch CUDA
Repeat train/holdout partitions with deterministic seeds to show score variation and the inflation caused by evaluating memorised examples.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download DGX Spark investigation ↓unzip y3-holdout-dgx-pathway.zip -d y3-holdout-dgx cd y3-holdout-dgx bash run-dgx.sh # Open the localhost notebook URL printed by the container.
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
A sealed-test protocol and a two-model comparison table.
Download the editable handout →