Building an evaluation set
On This Lesson 4 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
assemble a realistic, useful question set.
Planned topics
- Collect real questions (support tickets, logs, with personal data removed), not five hand-picked ones; start with about 50 and grow.
- Cover categories: normal questions, edge cases, questions that must be refused, adversarial ones.
- For each: the question, the expected source documents, and acceptance criteria (must mention X; must not claim Y).
- Version the set in Git and review it like code.
- Keep a held-out part you do not tune against.
Draft outline details
Demonstration
a row: {id: q17, question: "Can I return opened shoes?", sources: ["returns-policy#2"], must: ["30 days"], mustNot: ["free shipping"]}.
Check your understanding
Name three kinds of questions in a set.
Show answer
Normal, edge or adversarial, and must-refuse.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.