brightlab.Made for curious mindsBrightlab
Year 11 · Monitoring
How will we notice a deployed model is no longer behaving as expected?
Go to the investigation ↓A tested monitoring runbook with alert trace, rollback owner and recovery checks.
Take this investigation into Python: open the interactive Marimo notebook →
What should happen after a drift alarm?
Monitoring compares recent observations with a reference. This simulator advances a deterministic sensor stream with a known change point and optional short spike. A rolling mean can reveal sustained drift, but window length and alert threshold trade detection delay against false alarms. Detecting a changed input does not by itself show that prediction quality changed. A rollback needs a named owner and a known prior configuration.
Before this lesson: Compare a reference distribution with recent observations. Useful earlier investigations: Yesterday’s lagoon, today’s light; The cost of an alarm.
Monitoring compares recent observations with a reference. This simulator advances a deterministic sensor stream with a known change point and optional short spike. A rolling mean can reveal sustained drift, but window length and alert threshold trade detection delay against false alarms. Detecting a changed input does not by itself show that prediction quality changed. A rollback needs a named owner and a known prior configuration.
Compare a reference distribution with recent observations. Useful earlier investigations: Yesterday’s lagoon, today’s light; The cost of an alarm.
Plan two sessions and print the synthetic stream with the teacher’s change-point key hidden initially. Distinguish input monitoring from delayed outcome evaluation. Rehearse reset and stepping to reproduce the same stream.
ACARA V9 AI curriculum connection ↗ · Technologies ↗
A complete teaching sequence · 120 minutes
State the reference mean and who would receive an alert in the fictional system.
Ask: “Is an alert useful if nobody knows what to do?”
Listen for: “No; it needs an owner and a response.”
Predict delay and spike sensitivity for short versus long windows.
Ask: “What can smoothing remove along with noise?”
Listen for: “It can delay evidence of a real change.”
Step observations, inspect the rolling window and alert condition, then replay with a different window from the same start.
Ask: “Which observations contribute to today’s mean?”
Listen for: “Only those inside the current window.”
Use a brief anomaly without sustained drift and count false alerts.
Ask: “Does one unusual value establish a new normal?”
Listen for: “No; it might be temporary.”
In session two, select window, threshold and required persistence. Test spike and shift streams, measure detection delay and define investigation, rollback and recovery steps.
Ask: “When would an input alert justify rollback?”
Listen for: “After the policy’s evidence and safety checks, not automatically from every spike.”
Another group advances the stream and follows the documented policy.
Ask: “What do you record after recovery?”
Listen for: “Trigger, evidence, action, owner and checks that normal operation returned.”
Deployment is the end of testing.
A short spike can trigger an input alert without a lasting distribution change; a long window can miss an important change until late.
Create and replay a monitoring/rollback policy with explicit false-alarm and detection-delay evidence.
Ask for the exact observations in one rolling window and the first valid alert time. Check that a human owner and recovery criterion are specified.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| Monitoring mechanics | Reacts to every high point | Explains rolling mean and threshold | Quantifies false alarms and delay |
| Operational policy | Ends at an alert | Defines an owner and response | Tests rollback, recovery and limits of input-only evidence |
Use a ten-step stream and a three-point rolling average before exploring longer windows.
Compare mean-shift monitoring with an error-rate monitor using delayed labels; explain their different blind spots.
The stream is synthetic and no production service is controlled. Avoid treating a statistical alarm as evidence about a particular person. Senior mapping remains jurisdiction-specific.
Shared device? Turn remembering off. A project file lets you continue on another device.
Your browser is the laboratory
Increase the rolling window with the stream fixed. Predict effects on spike alerts and sustained-shift detection delay.
Reference mean is 10. Forty deterministic readings contain either a sustained +2 shift from step 21 or a brief +5 spike at step 15. Initial window is four readings and alert difference is 1. Only full windows can alert.
The experiment opens after your prediction.
There is no penalty for being surprised.
Apple Silicon · PyTorch MPS
Simulate thousands of independently noisy streams with a fixed known change point. Change the rolling window and compare detection-delay and false-alert quantiles.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download Mac Studio investigation ↓unzip y11-drift-mac-pathway.zip -d y11-drift-mac cd y11-drift-mac bash setup-mac.sh source .venv/bin/activate python experiment.py --device mps --output results marimo edit notebook.py
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
NVIDIA DGX Spark · PyTorch CUDA
Simulate thousands of independently noisy streams with a fixed known change point. Change the rolling window and compare detection-delay and false-alert quantiles.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download DGX Spark investigation ↓unzip y11-drift-dgx-pathway.zip -d y11-drift-dgx cd y11-drift-dgx bash run-dgx.sh # Open the localhost notebook URL printed by the container.
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
A tested monitoring runbook with alert trace, rollback owner and recovery checks.
Download the editable handout →