LLM-as-judge and its caveats
On This Lesson 4 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
use a model as a grader responsibly.
Planned topics
- A judge model scores answers against a rubric, scaling beyond what humans can read.
- Calibrate: have humans label a sample (for example 50), compare agreement with the judge, and adjust the rubric.
- Known biases: favors longer answers, favors its own style, can be inconsistent; use clear rubrics and fixed settings.
- Prefer deterministic checks where possible (citation present, source id matches).
- Re-calibrate when the judge model changes.
Draft outline details
Demonstration
agreement table: human label versus judge label on 50 samples, with the agreement rate.
Check your understanding
Why calibrate a judge?
Show answer
To know how far its scores match human judgment.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.