Model routing, caching, and streaming
On This Lesson 4 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
use routing, caching, streaming, and request shaping to hit targets.
Planned topics
- Routing: send easy requests to a smaller, cheaper model and hard ones to a larger one; measure quality per route.
- Caching: exact-match and semantic caches for repeated questions; mind freshness and privacy.
- Streaming: improves perceived latency by showing partial output early.
- Request shaping: shorter prompts, limited context, output limits, batching, and rate-limit-aware queues.
Draft outline details
Demonstration
a routing rule: order-status questions go to the small model with a data lookup; refund disputes go to the larger model.
Check your understanding
What must you measure when you route to a cheaper model?
Show answer
Quality on the evaluation set, not just cost.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.