Demo: an evaluation run catching a prompt regression
On This Lesson 3 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
watch the pipeline block a worse prompt. Prepares MID-070-LAB1.
Draft outline details
Step by step
- Run the 50-question set on the current prompt: baseline recorded.
- Open the report: recall, groundedness, and refusal correctness per category.
- Planned mistake: shorten the prompt to save tokens (dropping the "answer only from sources" rule).
- Run the pipeline: groundedness falls below the threshold; the build fails and lists the failing questions.
- Restore the rule, keep the shorter wording elsewhere, and rerun: pass.
- Open the dashboard: cost per conversation fell slightly, which would have hidden the regression.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.