AI vendor and model evaluation
On this page 9 topics
Overview
Topics
- AI vendor criteria
- public benchmarks vs own evaluations
- total cost
- exit plans
Learn first
3.5 h of practice (estimate)
Learning outcomes
- Define evaluation criteria for AI vendors and models: quality on your own tasks, latency, cost, data terms, security, support, and exit.
- Explain why public benchmarks are not enough and build a task-specific evaluation set.
- Run a time-boxed AI proof of concept that answers the riskiest questions.
- Compare total cost over several years, including usage growth and switching costs.
Lessons
Lab and exercises
Lab
ARC-047-LAB1 · AI vendor choice
Outcomes: 1–4 · Time: ~3.5 h
- Define gates, criteria, and weights for the Larkspur assistant with role-played stakeholders.
- Build a task-specific evaluation set (at least 50 questions including hard cases) and a scoring rubric.
- Run it against two hosted providers and one self-hosted open-weight model; record quality, latency, and cost per conversation. Verify model versions and prices before recording.
- Review each provider's terms (residency, retention, training use, exit) against Larkspur policy.
- Build a three-year cost model with usage growth scenarios and a switching reserve.
- Write a recommendation with evaluation scores, three-year cost, conditions, and an exit plan.
Acceptance: recommendation with evaluation scores, three-year cost, and exit plan.
Exercises (case-based; five exercises)
| ID | Outcome | Task |
|---|---|---|
| ARC-047-EX01 | 1 | List the criteria for choosing the Larkspur assistant's model provider, mark which are gates, and assign draft weights to the rest. |
| ARC-047-EX02 | 2 | The CIO proposes choosing the model that tops a public leaderboard. Write a short reply and describe what you would do instead. |
| ARC-047-EX03 | 2 | Design ten evaluation items from tickets, covering at least four categories and three hard cases. Show the fields recorded for each. |
| ARC-047-EX04 | 3 | Write a four-week PoC charter for the assistant with success criteria, tests, and decision owner. |
| ARC-047-EX05 | 4 | Larkspur expects 40,000 conversations a month in year one, growing 50% per year. Outline the three-year cost model structure for a hosted and a self-hosted option, including exit. Do not use real prices; mark them to verify. |
Show solutions
- EX01 Gates: data terms (no training on company data, retention and region acceptable), security controls. Weighted example: quality on own tasks 35%, cost 20%, latency 15%, support and roadmap 10%, exit and portability 10%, operating effort 10%; weights agreed with stakeholders before scoring.
- EX02 Leaderboards measure general public tasks; they do not reflect Larkspur's questions, documents, tone, latency, or cost. Build the evaluation set from tickets, run candidates, and decide on measured results with gates applied.
- EX03 Fields: id, question, category, source document, expected answer or rubric, difficulty, pass criteria. Categories such as order status, returns, delivery, loyalty; hard cases: out-of-scope, conflicting policy, injection attempt.
- EX04 Questions: accuracy grounded in documents, injection resistance, latency at expected load, cost per conversation; thresholds set in advance; real masked data; failure-mode tests; decision owner (e.g. head of customer service with security sign-off); exit if thresholds are not met.
- EX05 Conversations per month x cost per conversation (tokens in and out) per year with growth; hosted: usage cost, integration, evaluation, monitoring, people; self-hosted: hardware, operations staff, upgrades, plus usage; switching reserve and re-evaluation yearly; exit: portable prompts and evaluation set, abstraction layer, data return and deletion terms; prices to verify before recording.
All four outcomes are also covered by the lab acceptance and the quiz.
Workplace evidence: an AI vendor or model evaluation you ran and its outcome.
Quiz
Outcomes: O1 vendor and model criteria · O2 evaluations beyond benchmarks · O3 AI proofs of concept · O4 multi-year cost and exit.
A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.
Practical part
Evaluation report review on the Larkspur lab: criteria and weights, task-specific evaluation results for three approaches, terms review, three-year cost, and exit plan. Pass: quiz at least 8/10, at least four of five exercises correct, and a lab recommendation backed by evaluation scores, three-year cost, and an exit plan.
Common mistakes
Choosing on public leaderboard rank; PoCs on toy questions; ignoring data terms until contracting.
Assessment
Evaluation report review.
Workplace evidence
An AI vendor or model evaluation you ran and its outcome.
Career relevance
Vendor and procurement work in every AI-enabled solution.