Demo: scoring three AI vendors for Larkspur
On This Lesson 3 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
score two hosted providers and one self-hosted open-weight model for the Larkspur assistant. This prepares ARC-047-LAB1.
Draft outline details
Step by step
- Agree gates and weights with the role-played security officer and head of customer service: data terms and security as gates; quality, cost, latency, exit as weighted.
- Apply the gates from the terms review: one hosted provider needs a configuration change for retention; record it as a condition.
- Planned mistake: pick the provider with the top public leaderboard rank for the quality score. Fix: use the results from the evaluation set run on all three approaches.
- Fill the scoring matrix with measured quality, latency, estimated cost per conversation, and the operating effort for the self-hosted option.
- Run a weight sensitivity check: if cost is weighted higher, does the ranking change?
- Write the recommendation with conditions and the exit plan; verify prices and model versions before recording.
Check your understanding
What was the planned mistake?
Show answer
Using public leaderboard rank instead of results on the task-specific evaluation set.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.