Designing reliable AI features
On this page 9 topics
Overview
Topics
- Latency and cost budgets
- fallbacks
- provider outages
- model routing
- caching
- SLOs for non-deterministic features
Learn first
4 h of practice (estimate)
Learning outcomes
- Set latency, cost, and quality budgets for an LLM-backed feature and allocate them across retrieval, model calls, and tools.
- Design fallbacks for model or provider failure (smaller model, cached answer, non-AI path, honest "unavailable" state) that keep the core journey working.
- Use routing, caching, streaming, and request shaping to meet latency and cost targets.
- Define SLOs for non-deterministic features that combine availability, latency, and evaluated quality.
Lessons
- 01What makes AI features different to operate
- 02Latency and cost budgets for model calls
- 03Fallbacks and graceful degradation for AI
- 04Model routing, caching, and streaming
- 05SLOs for non-deterministic features
- 06Demo: an order-support assistant surviving a provider outage
- 07Demo: cutting cost per answer by routing easy questions to a smaller model
Lab and exercises
Lab
SRC-021-LAB1 · AI order support at 10×
Outcomes: 1–4 · Time: ~3 h
- Design the order-support assistant for the 10× forecast with p95 latency, cost-per-conversation, and quality targets, allocated across retrieval, model calls, and tools.
- Implement a fallback path and a model-routing rule.
- Define SLOs combining availability, latency, and evaluated quality.
- Inject a provider outage and a latency spike; run a load test; measure quality on the evaluation set before and after routing.
Acceptance: checkout and order tracking are unaffected by the outage; budgets met in a load test; quality measured on the evaluation set before and after routing.
Exercises (design review; six exercises)
| ID | Outcome | Task |
|---|---|---|
| SRC-021-EX01 | 1 | Allocate a 6 s p95 budget and a cost-per-conversation target across retrieval, model call, and tools for the assistant. |
| SRC-021-EX02 | 1 | Estimate monthly cost at 10x conversations given token counts and a price; state assumptions. |
| SRC-021-EX03 | 2 | Design a fallback ladder with triggers and user messages for model timeout, provider outage, and quota exhaustion. |
| SRC-021-EX04 | 2 | Review a design where the assistant call sits inside the checkout transaction; fix it. |
| SRC-021-EX05 | 3 | Propose a routing rule and a cache for order-status and refund questions; list the risks of each. |
| SRC-021-EX06 | 4 | Write SLOs for the assistant including a quality SLI, and respond to an assessor constraint: cost per answer must be halved. |
Show solutions
- EX01 For example retrieval 600 ms, prompt assembly 100 ms, model 4 s, tools 600 ms, margin 700 ms; cost split by tokens; include timeouts per stage.
- EX02 Conversations x turns x (input + output tokens) x price per token; verify current pricing; include retrieval and retries; show a range.
- EX03 Timeout: retry once then smaller model; outage: circuit opens, cached or FAQ answer, then honest "unavailable"; quota exhaustion: queue with limits, route to cheaper model, shed low-priority traffic.
- EX04 Move it off the critical path: asynchronous endpoint, separate pool, timeout; checkout must not depend on model availability.
- EX05 Routing: order-status via data lookup and small model, refunds to the large model; risk of misrouting measured on the evaluation set; cache exact-match answers with short TTL, avoid caching personal data across users.
- EX06 SLOs: 99.5% of conversations get a response in 6 s p95, 90% of sampled answers meet the rubric, cost under budget. To halve cost: more routing to small model, shorter context, caching, output limits; re-evaluate quality and report the trade-off.
Workplace evidence: an AI feature you designed or hardened with explicit latency, cost, and fallback targets.
Quiz
Outcomes: O1 latency, cost, quality budgets · O2 fallbacks · O3 routing, caching, streaming · O4 SLOs for non-deterministic features.
A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.
Practical part
Six design exercises SRC-021-EX01 to EX06 plus the lab with an injected provider outage and latency spike. Pass: quiz at least 8/10, at least five exercises correct, and a design review that holds up under an assessor-injected constraint (cost halved or quota cut).
Common mistakes
Putting a model call on the critical checkout path; no timeout on model calls; measuring only availability for an AI feature; assuming the provider's rate limits will scale with you.
Assessment
Design review of the assistant with an assessor-injected constraint (cost halved, or the provider's quota cut).
Workplace evidence
An AI feature you designed or hardened with explicit latency, cost, and fallback targets.
Career relevance
AI platform capabilities (SRB-021) and AI solution architecture (ARC-045).