Demo: building a 50-question evaluation set from support tickets
On This Lesson 3 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
build a first 50-question evaluation set from Larkspur support tickets. This prepares ARC-047-LAB1.
Draft outline details
Step by step
- Sample recent tickets across categories (order status, returns, delivery, store opening hours, loyalty), with privacy approval.
- Remove personal data (names, addresses, card digits) and rewrite each into a clean question with the information the assistant would have.
- Planned mistake: write all 50 questions that the knowledge base answers easily. Fix: add about a fifth as hard cases: out-of-scope, ambiguous, conflicting policy, a prompt-injection attempt, and a request needing a human.
- For each question record the expected answer or rubric and the source document.
- Have two reviewers check a sample for agreement; resolve disagreements and fix unclear rubrics.
- Store the set versioned and private, run the baseline, and record the results for use in the lab.
Check your understanding
What was the planned mistake?
Show answer
Including only easy questions and no hard or out-of-scope cases.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.