Latency and cost budgets for model calls
On This Lesson 4 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
budget latency, cost, and quality across retrieval, model, and tools.
Planned topics
- Budget end to end: retrieval, prompt assembly, model time to first token and total, tool calls, post-processing.
- Cost per conversation: tokens in and out times price, plus retrieval and tool costs (verify current pricing before recording).
- Set a quality target measured on an evaluation set; budgets trade against quality.
- Every model call gets a timeout, a token limit, and a retry policy.
Draft outline details
Demonstration
a budget table: p95 latency, cost per conversation, and quality score per stage of the order-support assistant.
Check your understanding
What must every model call have?
Show answer
A timeout, a token limit, and a retry or fallback policy.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.