Organization-wide reliability
On this page 9 topics
Overview
Topics
- Reliability programs
- systemic risk reduction
- operational maturity
Learn first
4 h of practice (estimate)
Learning outcomes
- Find systemic reliability risks from incident patterns across teams.
- Run a reliability program: SLO coverage, postmortem quality, action-item follow-through, on-call health.
- Assess operational maturity of teams and target help where it matters most.
- Report reliability to leadership as business risk with trends.
Lessons
- 01Reliability as an organizational property
- 02Mining incidents for systemic causes
- 03Designing a reliability program
- 04Operational maturity models
- 05On-call health
- 06Reporting reliability to leadership
- 07Demo: finding the cause behind 30 incidents in 12 teams
- 08Demo: a quarterly reliability review with the CTO
Lab and exercises
Lab
STF-005-LAB1 · Reliability program
Outcomes: 1–4 · Time: ~3 h
- Take a year of Platform 2.0 incident reports (case materials) and tag them with a common scheme.
- Identify three systemic causes with counts and affected teams.
- Design the program: SLO coverage, postmortem quality, action follow-through, on-call health, each with a target and owner.
- Add a maturity self-assessment to direct help.
- Draft a one-page leadership report with trends and requested decisions.
Acceptance: program targets are measurable and tied to the systemic causes.
Exercises (case-based; six exercises)
| ID | Outcome | Task |
|---|---|---|
| STF-005-EX01 | 1 | Given a table of 20 incidents with causes, group them into systemic and one-off causes and justify. |
| STF-005-EX02 | 1 | A postmortem says "engineer ran the wrong command". Rewrite the finding to expose the systemic cause and the fix. |
| STF-005-EX03 | 2 | Set targets for SLO coverage, action-item follow-through, and postmortem quality; define how each is measured. |
| STF-005-EX04 | 3 | Design a five-practice maturity grid and show how you would use it to pick three teams for help. |
| STF-005-EX05 | 3 | Review on-call data (pages per week, share actionable) for four rotations and recommend actions in priority order. |
| STF-005-EX06 | 4 | Write a one-page quarterly summary for the CTO translating reliability data into business risk and two decisions. |
Show solutions
- EX01 Systemic: recurring across several teams (rollback gaps, alert gaps). One-off: single team, unique trigger. Show counts.
- EX02 Missing guardrail: confirmation or environment labeling, access limits, automation; fix is a safer tool, not blame.
- EX03 Examples: 80% of tier-one services with SLOs in two quarters; 90% of actions closed within 60 days; postmortems reviewed against a template with contributing factors. Measure from the SLO catalog and the tracker.
- EX04 Practices: monitoring, SLOs, tested rollback, runbooks, game days. Pick teams that own critical paths and score lowest.
- EX05 Fix highest noise first, then rotation size, then runbooks; escalate rotations above the paging target.
- EX06 Show revenue-minutes at risk, trend arrows, top three risks, cost of the investment asked for.
Outcome coverage: all four outcomes are covered by the exercises and lab.
Workplace evidence: a reliability program or systemic fix that reduced repeat incidents across teams.
Quiz
Outcomes: O1 systemic risks from incidents · O2 reliability program · O3 operational maturity · O4 reporting reliability as business risk.
A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.
Practical part
Six case exercises STF-005-EX01 to EX06 plus the lab, reviewed as a reliability program. Pass: at least five of six exercises correct, quiz at least 8/10, and a program whose targets are measurable and tied to the three systemic causes.
Common mistakes
Fixing incidents one by one; postmortem actions that are never done; equating reliability with uptime alone.
Assessment
Program review.
Workplace evidence
A reliability program or systemic fix that reduced repeat incidents across teams.
Career relevance
Crisis leadership (STF-018) and resilience architecture (ARC-031).