0110 min
Set the deadline
Place feature cards before or after the prediction time on a timeline.
Ask: “Could the system know the inspection result before the event?”
Listen for: “No, it is collected later.”
Brightlab
brightlab.Made for curious mindsbrightlab.Made for curious mindsBrightlab
Year 7 · Evaluation
Is a brilliant score hiding information from the future?
Go to the investigation ↓A timestamped feature contract and a leaky/time-valid evaluation report.
Was this feature available when the prediction had to be made?
A useful predictor may use only information available when the prediction is made. A post-event measurement can reveal the outcome and make evaluation look excellent. This lab compares a valid pre-event signal with a synthetic post-event indicator. Its high leaky score is real computation on unsuitable inputs. Removing leakage usually lowers apparent performance while making the evaluation more relevant.
Before this lesson: Separate training and testing and order events by time. Useful earlier investigations: The sealed test envelope.
A useful predictor may use only information available when the prediction is made. A post-event measurement can reveal the outcome and make evaluation look excellent. This lab compares a valid pre-event signal with a synthetic post-event indicator. Its high leaky score is real computation on unsuitable inputs. Removing leakage usually lowers apparent performance while making the evaluation more relevant.
Separate training and testing and order events by time. Useful earlier investigations: The sealed test envelope.
Print the synthetic tide-event table with timestamps for forecast readings and post-event inspection. Decide the prediction deadline before showing scores. No actual tide forecast is used.
ACARA V9 AI curriculum connection ↗ · Technologies ↗
A complete teaching sequence · 75 minutes
Place feature cards before or after the prediction time on a timeline.
Ask: “Could the system know the inspection result before the event?”
Listen for: “No, it is collected later.”
Predict which model scores higher before enabling the post-event feature.
Ask: “Why might the wrong feature make a score look better?”
Listen for: “It nearly contains the answer.”
Toggle the leaked feature while keeping events fixed. Inspect features, predictions and error counts. Record denominators.
Ask: “Is the calculation fake, or is the experiment invalid for the intended use?”
Listen for: “The calculation is real, but the input would not be available.”
Inspect the edge case in which the post-event feature has an innocent name such as final flag.
Ask: “Can a harmless column name make the feature valid?”
Listen for: “No; the timestamp and origin matter.”
Exclude late features, state a baseline and write a schema that records availability time. Re-evaluate and narrow the performance claim.
Ask: “Why is a lower honest score useful?”
Listen for: “It tells us what the system can really do at prediction time.”
A partner proposes a derived feature; decide whether its ingredients cross the boundary.
Ask: “Can a transformed late feature still leak?”
Listen for: “Yes, transforming it does not change when it became available.”
A very high test score always means a useful model.
A renamed or transformed post-event flag still leaks future information even though its name looks harmless.
Create a feature-availability schema and evaluate a time-valid model against a declared baseline.
Ask learners to trace the origin of a derived field. Credit a lower score with a valid boundary over an unjustified perfect score.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Leak diagnosis | Trusts the largest score | Identifies a post-event input | Traces leakage through derived fields |
| Evaluation repair | Deletes columns without a rule | Enforces a stated prediction deadline | Compares a baseline and limits the resulting claim |
Use a physical before/after timeline and four events before examining the full table.
Design a time-ordered split and explain why random row splitting can still leak repeated-event information.
Synthetic tide events only. These models are not forecasts and must not inform coastal safety decisions.
Shared device? Turn remembering off. A project file lets you continue on another device.
Your browser is the laboratory
Predict how adding a post-event indicator changes accuracy without changing the test events.
Sixteen synthetic events have a pre-event signal and a post-event inspection flag. The inspection flag exactly matches the outcome, but arrives after the prediction deadline. Both models use the same test events.
The experiment opens after your prediction.
There is no penalty for being surprised.
Apple Silicon · PyTorch MPS
Evaluate thousands of synthetic events using a pre-event feature, a copied post-event outcome and a fixed baseline. Compare measured accuracy with whether the feature existed at prediction time.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download Mac Studio investigation ↓unzip y7-leakage-mac-pathway.zip -d y7-leakage-mac cd y7-leakage-mac bash setup-mac.sh source .venv/bin/activate python experiment.py --device mps --output results marimo edit notebook.py
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
NVIDIA DGX Spark · PyTorch CUDA
Evaluate thousands of synthetic events using a pre-event feature, a copied post-event outcome and a fixed baseline. Compare measured accuracy with whether the feature existed at prediction time.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download DGX Spark investigation ↓unzip y7-leakage-dgx-pathway.zip -d y7-leakage-dgx cd y7-leakage-dgx bash run-dgx.sh # Open the localhost notebook URL printed by the container.
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
A timestamped feature contract and a leaky/time-valid evaluation report.
Download the editable handout →