Operating AI in production
On this page 9 topics
Overview
Topics
- Prompt and model versioning and rollback
- evaluation-gated releases
- drift
- bad-output incidents
- human review
Learn first
4 h of practice (estimate)
Learning outcomes
- Version prompts, model choices, and retrieval settings as deployable configuration with rollback.
- Gate releases on evaluation results and roll out prompt or model changes progressively.
- Detect quality drift (model updates, data changes, new user behavior) from evaluation and production signals.
- Run incidents for bad AI output (wrong, unsafe, or leaking answers), including kill switches and human review queues.
Lessons
- 01AI changes are deployments too
- 02Versioning prompts, models, and retrieval
- 03Evaluation-gated and progressive releases
- 04Detecting drift
- 05Bad-output incidents, kill switches, and human review
- 06Demo: rolling back a prompt change that broke refund answers
- 07Demo: an incident where the assistant revealed another customer's order
Lab and exercises
Lab
SRC-022-LAB1 · Run the assistant
Outcomes: 1–4 · Time: ~4 h Put the order-support assistant's prompts and model choice under versioned configuration. Add an evaluation gate and a canary. Rehearse a bad-output incident with a kill switch and a human review queue, then write the postmortem. Acceptance: a seeded regression is blocked by the gate; the kill switch disables the feature in under one minute; postmortem written.
Exercises (assessment: incident simulation on a misbehaving AI feature)
| ID | Outcome | Task |
|---|---|---|
| SRC-022-EX01 | 1 | Design the configuration bundle for the assistant: what is versioned, what is pinned, and what is logged per answer. |
| SRC-022-EX02 | 1 | Case: the team edits the prompt in a production console. Write the review comment and the process replacing it. |
| SRC-022-EX03 | 2 | Specify an evaluation gate: case categories, per-category thresholds, hard safety cases, and promote/abort rules for a canary. |
| SRC-022-EX04 | 3 | Case: answers worsen in November with no deploy. List four hypotheses and the signal that tests each. |
| SRC-022-EX05 | 4 | Write the kill-switch design and a runbook for "assistant gave wrong refund advice to many customers". |
| SRC-022-EX06 | 4 | Review an incident timeline where the fix was a prompt warning not to leak data. Explain why it is insufficient and propose the real control. |
Show solutions
- EX01 Version: prompt, model id (pinned), parameters, retrieval index and settings, eval set. Log: bundle ID, model version, retrieved source IDs, request ID.
- EX02 Direct edits skip review, tests and rollback. Replace with config in version control, PR review, eval gate, staged rollout, audit log.
- EX03 Categories (refunds, shipping, account); overall and per-category thresholds relative to baseline; zero tolerance on leakage and unsafe cases; promote after bake with stable escalation and thumbs-down rates.
- EX04 Model updated by provider (compare pinned version, rerun eval), knowledge data changed (index diff), seasonal topics (topic clusters), new user behavior (sampled grading). Each tested by scheduled eval or sampling.
- EX05 Flag checked on each request, falling back to a safe path such as handing off to humans; tested regularly and timed under one minute. Runbook: detect, kill, contain, identify affected customers, communicate, root cause, add eval case, postmortem.
- EX06 Prompts are advisory and can be bypassed; enforce authorization in the retrieval and tool layer using the caller's identity, and add cross-customer eval cases.
- Outcome 2 is also covered by the lab gate and Q04-Q05.
Workplace evidence: a release process or incident you ran for an AI feature.
Quiz
Outcomes: O1 versioned configuration · O2 gated and progressive release · O3 drift · O4 bad-output incidents.
A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.
Practical part
Incident simulation on a misbehaving AI feature (SRC-022 assessment), plus SRC-022-LAB1 evidence. Pass: at least 8/10 on the quiz, kill switch under one minute, a seeded regression blocked by the gate, and a postmortem with systemic actions.
Common mistakes
Editing prompts directly in production; no record of which prompt and model produced an answer; treating unsafe output as "just a bug" without an incident process.
Assessment
Incident simulation on a misbehaving AI feature.
Workplace evidence
A release process or incident you ran for an AI feature.
Career relevance
Shared AI platforms (SRB-021) and responsible AI controls (SRA-022).