Demo: costing the assistant per conversation
On This Lesson 3 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
estimate cost and latency per conversation, not per request, and design fallbacks.
Draft outline details
Step by step
- Gather volumes with customer service: conversations per month, turns per conversation, share that hand over.
- Estimate tokens per turn: system prompt, retrieved text, history, and answer; the history grows each turn.
- Multiply by the provider's price per input and output token (verify current prices before recording) to get cost per conversation.
- Planned mistake: cost one request and multiply by conversations. Fix: sum all turns, including retrieval and retries, then add vector search and hosting costs.
- Estimate latency end to end: retrieval, tool calls, model; set a timeout.
- Define fallbacks: after timeout show a static help page; during provider outage route to the human queue; compare cost against a human-handled conversation.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.