Demo: tuning noisy alerts with burn-rate windows
On This Lesson 3 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
replace noisy threshold alerts with burn-rate alerts; prepares SRC-012-LAB1.
Draft outline details
Step by step
- Show the current alert (error rate above 1% for 1 minute) and last month's pages: most were blips.
- Replay recorded error data against the rule and count pages versus real incidents.
- Define burn-rate rules with long and short windows for the 99.95% SLO.
- Replay again: blips stay silent, the two real outages page quickly.
- Planned mistake: use only a long window, so the page keeps firing long after recovery. Fix: add the short confirming window.
- Record precision and recall before and after.
Check your understanding
Which two numbers measure alert quality?
Show answer
Precision and recall.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.