Evaluating AI vendors and models
On This Lesson 4 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
define the criteria for choosing an AI vendor or model and weight them with stakeholders.
Planned topics
- Core criteria: quality on your own tasks, latency, cost, data terms (residency, retention, training use), security, support and roadmap, and exit.
- Separate vendor questions (terms, support, stability, sub-processors) from model questions (quality, speed, context size, tool use).
- Agree weights with stakeholders before scoring; security and data terms may be pass/fail gates, not weighted points.
- Include operating effort: a self-hosted open-weight model needs GPUs, patching, and skilled people.
- Reuse the vendor-evaluation method from ARC-040; the AI-specific part is how quality is measured.
Draft outline details
Demonstration
a criteria table for Larkspur: criterion, gate or weighted, weight, how measured, evidence source; two hosted providers and one self-hosted open-weight model as columns.
Check your understanding
Which criteria might be gates instead of weighted points?
Show answer
Data terms and security requirements, which must be met regardless of score.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.