brightlab.Made for curious mindsBrightlab
Year 10 · Retrieval
When should an assistant answer, and when should it stop?
Go to the investigation ↓An extractive assistant system card with source-specific traces and an answerability test suite.
Take this investigation into Python: open the interactive Marimo notebook →
When should an answer be supported, qualified or withheld?
This local assistant retrieves a supplied passage using keyword overlap, then applies an explicit evidence gate. It returns a quoted extract only when the required fact is present and the score meets a threshold; otherwise it refuses or requests clarification. No language model generates prose. Visible retrieval and refusal make a small system inspectable, but neither a score nor a citation can guarantee that the source is current or true.
Before this lesson: Match claims to evidence and inspect retrieved passages. Useful earlier investigations: A claim needs somewhere to stand; Where should a document break?.
This local assistant retrieves a supplied passage using keyword overlap, then applies an explicit evidence gate. It returns a quoted extract only when the required fact is present and the score meets a threshold; otherwise it refuses or requests clarification. No language model generates prose. Visible retrieval and refusal make a small system inspectable, but neither a score nor a citation can guarantee that the source is current or true.
Match claims to evidence and inspect retrieved passages. Useful earlier investigations: A claim needs somewhere to stand; Where should a document break?.
Read the fictional field notes and four supplied queries: supported, absent, ambiguous and conflicting. Rehearse source removal and threshold control. Explain that this deterministic assistant is simpler than an LLM-based RAG system.
ACARA V9 AI curriculum connection ↗ · Technologies ↗
A complete teaching sequence · 85 minutes
Define that answers must come from the supplied notes, including dates and conditions.
Ask: “What should happen when the notes do not say?”
Listen for: “The helper should say the evidence is missing.”
Classify the four queries before running them.
Ask: “Is a similar topic enough to answer the exact question?”
Listen for: “No, the required fact must be present.”
Edit a query, inspect term matches, top passage and answerability check, then compare thresholds with sources fixed.
Ask: “Which stage rejected this query?”
Listen for: “Retrieval was weak, or the required fact was absent.”
Disable the relevant note while keeping the query unchanged.
Ask: “Should the assistant repeat its earlier answer from memory?”
Listen for: “No; it must reflect the current supplied evidence.”
Choose a threshold, run all four cases and record supported answers, false answers, refusals and false refusals. Add one adversarially phrased question.
Ask: “Could a safer threshold become too unhelpful?”
Listen for: “Yes, it can reject answerable questions.”
Document sources, gate, evaluation and limits.
Ask: “What does this lab actually compute?”
Listen for: “Lexical retrieval, explicit checks and extractive responses.”
Retrieval eliminates invented answers.
A high-scoring passage can be about the right topic while lacking the required fact; removing a source must change answerability.
Specify an evidence gate and evaluate supported answers and false refusals on the fixed suite plus one new challenge.
Require the retrieved passage and gate reason for every reported result. Check that a citation is not treated as automatic proof of a generated claim.
| Criterion | Beginning | Secure | Extending |
|---|---|---|---|
| System trace | Treats the helper as one black box | Separates retrieval, gate and response | Diagnoses the first failing stage |
| Evaluation | Counts only successful answers | Tests missing and ambiguous evidence | Reports false answers, false refusals and residual source risk |
Begin with fixed query buttons alongside an editable query field, then compose a new question with a partner.
Add a second conflicting dated source and design a conflict response rather than arbitrarily selecting the highest score.
No external model or API is called. Use only synthetic notes; do not paste student information or confidential school policies.
Shared device? Turn remembering off. A project file lets you continue on another device.
Your browser is the laboratory
Predict supported answer, missing evidence or ambiguity for each supplied query before running the assistant.
Source A: gate opens at 8 am on dry days, closed on wet days. B: Saturday walks at 10 am, 12 places. C: free paper map at the desk. Initial question asks when the reserve gate opens. No swimming or future-weather evidence is supplied.
The experiment opens after your prediction.
There is no penalty for being surprised.
Apple Silicon · PyTorch MPS
Evaluate thousands of structured synthetic answerability cases. Sweep retrieval thresholds and compare false answers and refusals with and without a required-fact gate; relevance scores are supplied fixtures.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download Mac Studio investigation ↓unzip y10-grounding-mac-pathway.zip -d y10-grounding-mac cd y10-grounding-mac bash setup-mac.sh source .venv/bin/activate python experiment.py --device mps --output results marimo edit notebook.py
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
NVIDIA DGX Spark · PyTorch CUDA
Evaluate thousands of structured synthetic answerability cases. Sweep retrieval thresholds and compare false answers and refusals with and without a required-fact gate; relevance scores are supplied fixtures.
Students extend the experiment in teams, documenting parameters, outputs and limitations.
The bundle contains lesson-specific working code, a configuration file, a reactive notebook, a deterministic CPU check and hardware setup instructions. Acceleration is reported only after a tensor operation and result read-back succeed.
Download DGX Spark investigation ↓unzip y10-grounding-dgx-pathway.zip -d y10-grounding-dgx cd y10-grounding-dgx bash run-dgx.sh # Open the localhost notebook URL printed by the container.
CPU and available-device execution status is recorded in the downloaded README and validation report. DGX Spark execution requires that hardware; static validation alone does not establish GPU compatibility or performance. The browser lesson remains fully available without this extension.
An extractive assistant system card with source-specific traces and an answerability test suite.
Download the editable handout →