انتقل إلى المحتوى
أمين أبوزيد
Englishالعربية
MID-070

تقييم LLM وقابلية رصده

  • 7 درس
  • 3 ساعة تدريب (تقدير)
ابدأ الوحدة
في هذه الصفحة 8 موضوعات

نص الدروس والتمارين باللغة الإنجليزية، بينما تبقى واجهة الموقع وعناوين الدروس بالعربية.

نظرة عامة

الموضوعات

  • مجموعات التقييم في CI
  • محاذير استخدام LLM كحَكَم
  • كشف التراجعات
  • تتبّع الرموز (tokens)
  • زمن الاستجابة
  • والكلفة

تعلّم أولًا

3 ساعة تدريب (تقدير)

مخرجات التعلم

  1. Build an evaluation set of realistic questions with expected sources and acceptance criteria.
  2. Measure retrieval recall, answer groundedness (faithfulness to sources), and refusal correctness.
  3. Use an LLM as a judge with calibration against human-labeled samples, and state its limits.
  4. Run evaluations in CI so prompt, model, or retrieval changes cannot silently degrade quality.
  5. Trace each request with token counts, latency per step, model version, and cost, and build dashboards and alerts on them.

الدروس

  1. 01لماذا تحتاج ميزات LLM إلى تقييمات
  2. 02بناء مجموعة تقييم
  3. 03المقاييس: الاستدعاء (recall) والارتكاز على المصادر (groundedness) والرفض
  4. 04LLM كحَكَم ومحاذيره
  5. 05التقييمات في CI
  6. 06تتبّع الرموز وزمن الاستجابة والكلفة
  7. 07عرض عملي: تشغيل تقييم يكشف تراجعًا في الـ prompt

المختبر والتمارين

Lab

MID-070-LAB1 · Measured assistant

Outcomes: 1–5 · Time: ~3 h Create a 50-question evaluation set for the order-support assistant and run it in CI with pass thresholds. Add OpenTelemetry spans for retrieval, model call, and tools with token and cost attributes. Build a Grafana dashboard for cost per conversation and p95 latency. Acceptance: a deliberately worse prompt fails the pipeline; cost per conversation is visible per day.

Exercises (assessment: evaluation plan for a new AI feature)

IDOutcomeTask
MID-070-EX011Write 20 evaluation rows (question, expected sources, must and must-not criteria) covering normal, edge, adversarial, and must-refuse cases.
MID-070-EX022Compute retrieval recall for the set by checking whether the expected source is in the top-k results.
MID-070-EX032Write a rule-based check for refusal correctness and one for citation presence.
MID-070-EX043Label 30 answers by hand, run a judge model with a rubric, and compute agreement; list two judge failures.
MID-070-EX054Add a CI stage that fails when groundedness drops below 0.9 or falls more than 3 points from baseline.
MID-070-EX065Add spans with model version, prompt version, tokens, and cost; verify them in the trace backend.
MID-070-EX071–5Design an evaluation plan for a new "product description generator": data, metrics, thresholds, what blocks a release, and the dashboards and alerts.
أظهر الحلول
  • EX01 Include rows such as a policy question, an ambiguous one, a prompt-injection attempt, and one whose answer is not in the sources (expected: refusal).
  • EX02 recall = rows where expected source in top-k / total rows.
  • EX03 Refusal: the answer matches a refusal pattern exactly when no source supports it; citation: the answer cites a retrieved source id.
  • EX04 Agreement = matching labels / 30; typical failures: rewards verbose answers, misses subtle unsupported claims.
  • EX05 Compare the run's metrics with the stored baseline; non-zero exit when below threshold or margin.
  • EX06 Span attributes for model, prompt version, token counts, and computed cost; check them in Jaeger or Tempo.
  • EX07 Include a held-out set, human review sample, calibrated judge, thresholds (for example groundedness at least 0.9), release blockers (any threshold miss, any safety failure), and cost and latency alerts.

Outcome 1 is also checked by quiz Q01-Q02; outcome 3 by the quiz and EX04.

الاختبار

Outcomes: O1 evaluation sets · O2 metrics · O3 LLM-as-judge · O4 evaluations in CI · O5 tracing tokens and cost.

اختبار ذاتي لك: لا توجد درجات، ولا يُحفظ أو يُرسل أي شيء. اختر إجابة ثم تحقق منها، ويمكنك المحاولة مرة أخرى كما تشاء.

  1. 1Why use an evaluation set?O1
  2. 2A good evaluation set includes:O1
  3. 3Retrieval recall measures whether:O2
  4. 4Groundedness means:O2
  5. 5Refusal correctness means the assistant:O2
  6. 6Before trusting a judge model you should:O3
  7. 7A known bias of judge models:O3
  8. 8In CI the evaluation should fail the build when:O4
  9. 9Which belongs on a model-call span?O5
  10. 10Cost problems are best caught by:O5

الجزء العملي

Seven exercises MID-070-EX01 to EX07 (EX07 is the evaluation plan for a new AI feature) plus the measured-assistant lab. Pass: at least five exercises including EX07, and a lab where a worse prompt fails the pipeline and cost per conversation is visible per day.

أخطاء شائعة

Evaluating with five hand-picked questions; trusting an uncalibrated judge model; no record of which prompt and model version produced an answer; cost discovered on the monthly invoice.

التقييم

Design an evaluation plan for a new AI feature, including thresholds and what blocks a release.

الصلة بالمسار الوظيفي

Evaluation-gated releases (SRC-022) and AI vendor and model evaluation (ARC-047).

اكتب للبحث في المستويات والمراحل والوحدات والدروس.