brightlab.Made for curious mindsBrightlab
Year 8 · Evaluation
When does a better fit become a worse prediction?
Go to the investigation ↓A capacity-selection report with residual sketches and a sealed-test decision record.
Does the curve explain the pattern or remember the noise?
A polynomial can bend more as its degree increases. Fitting minimises squared error on the training points, including their noise. A flexible curve may follow that noise and predict poorly between points. We use a separate validation set to choose degree and keep a final test sealed until the choice is made. This is a computed polynomial fit to synthetic data, not a claim about all models.
Before this lesson: Read a coordinate plot and distinguish training from testing. Useful earlier investigations: The sealed test envelope.
A polynomial can bend more as its degree increases. Fitting minimises squared error on the training points, including their noise. A flexible curve may follow that noise and predict poorly between points. We use a separate validation set to choose degree and keep a final test sealed until the choice is made. This is a computed polynomial fit to synthetic data, not a claim about all models.
Read a coordinate plot and distinguish training from testing. Useful earlier investigations: The sealed test envelope.
Review squared residuals. Print the fixed training and validation points in different symbols. Rehearse degree control, lock choice and final-test reveal; explain that reset begins a new classroom demonstration, not a newly independent dataset.
ACARA V9 AI curriculum connection ↗ · Technologies ↗
A complete teaching sequence · 75 minutes
Sketch a simple curve through noisy points and compare with a curve touching every point.
Ask: “Must every wobble belong to the underlying pattern?”
Listen for: “Some might be measurement noise.”
Predict training and validation changes from degree 1 to a high degree.
Ask: “Could one error fall while another rises?”
Listen for: “Yes, if the curve starts fitting noise.”
Adjust degree, inspect the fitted curve and separate error summaries. Record three degrees with data fixed.
Ask: “Where are the largest validation residuals?”
Listen for: “Often between training points where the curve bends sharply.”
Use the clean-data edge case to show that extra capacity is not always harmful.
Ask: “Is a higher degree automatically wrong?”
Listen for: “No, it depends on data and the underlying function.”
Choose degree using validation, write a reason and lock it before revealing final-test results. Report any gap without retuning the reported final result.
Ask: “What happens if we use the final score to choose again?”
Listen for: “It becomes another validation set.”
Submit choice, training/validation comparison and one final-test result.
Ask: “What does this test fail to tell us?”
Listen for: “Performance outside the sampled range or under new conditions.”
The most detailed fit is the best model.
On clean data generated by a higher-order curve, extra capacity can help; degree alone does not diagnose overfitting.
Select degree from validation evidence, lock it and report the sealed final test once.
Check the timing of the degree decision and final-test reveal. Ask for a specific residual supporting the overfitting explanation.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Fit interpretation | Prefers the smallest training error | Uses the validation gap | Explains residual geometry and a counterexample |
| Selection protocol | Tunes repeatedly on final results | Locks a validation-based choice | Reports final uncertainty and avoids inflated claims |
Compare only degree 1, 3 and 7 and use precomputed residual squares before exploring all degrees.
Add regularisation and compare capacity reduction with shrinking coefficients.
All curves are synthetic. No predictions about students’ grades or abilities are modelled.
Shared device? Turn remembering off. A project file lets you continue on another device.
Your browser is the laboratory
Increase degree with all points fixed. Predict whether training and validation errors move together.
Nine training points sample a quadratic with deterministic noise; twelve validation points are separate. Fit degrees 1–7, choosing on validation. Thirteen final-test points remain sealed until degree is locked.
The experiment opens after your prediction.
There is no penalty for being surprised.
Apple Silicon · PyTorch MPS
Fit seven polynomial degrees to fixed noisy training data on CPU, then evaluate thousands of unseen synthetic points on MPS/CUDA. Change ridge regularisation and compare training and unseen error.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download Mac Studio investigation ↓unzip y8-fit-mac-pathway.zip -d y8-fit-mac cd y8-fit-mac bash setup-mac.sh source .venv/bin/activate python experiment.py --device mps --output results marimo edit notebook.py
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
NVIDIA DGX Spark · PyTorch CUDA
Fit seven polynomial degrees to fixed noisy training data on CPU, then evaluate thousands of unseen synthetic points on MPS/CUDA. Change ridge regularisation and compare training and unseen error.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download DGX Spark investigation ↓unzip y8-fit-dgx-pathway.zip -d y8-fit-dgx cd y8-fit-dgx bash run-dgx.sh # Open the localhost notebook URL printed by the container.
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
A capacity-selection report with residual sketches and a sealed-test decision record.
Download the editable handout →