Evaluating retrieval separately from generation
On This Lesson 4 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
measure retrieval quality apart from answer quality.
Planned topics
- Two stages, two questions: did we retrieve the right chunk? Did the model use it correctly?
- Build a question set with the expected article or chunk for each question.
- Retrieval metrics: recall@k (the right chunk is in the top k) and, optionally, rank of the first right chunk.
- Check retrieval first: if recall is low, no prompt change will help.
- Fluent answers prove nothing; judge by sources and the expected content (answer evaluation is covered in MID-070).
Draft outline details
Demonstration
table: 30 questions, recall@4 = 27/30 = 0.90; the 3 misses listed with their cause (wrong language, chunk too large, missing synonym).
Check your understanding
Why measure retrieval separately?
Show answer
To find whether failures come from search or from generation, and fix the right layer.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.