Detecting drift
On This Lesson 4 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
notice quality decay when nothing in your code changed.
Planned topics
- Sources of drift: provider model updates, changes in the knowledge data, new kinds of user questions, seasonality.
- Run the evaluation set on a schedule against production configuration; alert on score drops.
- Sample live conversations for human or automated grading and watch distributions of topics and refusal rates.
- Pinning model versions where possible reduces surprise; verify what your provider guarantees about deprecation.
Draft outline details
Demonstration
A dashboard sketch: nightly eval score line, live sampled-grade line, escalation rate, new-topic cluster count, with alert thresholds.
Check your understanding
Name two causes of drift with no code change.
Show answer
For example a model update and changes in the data or user questions.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.