Cascading and metastable failures
On This Lesson 4 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
recognize failure patterns that sustain themselves and break them.
Planned topics
- Cascading failure: overload in one component pushes load to others (failover traffic, retries) until many fail.
- Metastable failure: a trigger (a short slowdown) leaves the system in a bad state sustained by its own reaction, such as retries or queue backlog, even after the trigger is gone.
- Retry storm: clients retry failing calls, multiplying load; layered retries multiply (3 × 3 × 3 = 27 calls).
- Breaking the loop: shed load, disable or limit retries, circuit breakers, backoff with jitter, drain queues or reduce inflow, then restore capacity gradually.
- Prevent: retry budgets, timeouts that shrink down the call chain, admission control.
Draft outline details
Demonstration
Diagram description: slow database leads to timeouts, which cause retries, which increase load, which slows the database further; a loop with the trigger removed but the loop still running.
Check your understanding
What makes a failure metastable?
Show answer
The system's own reaction, such as retries or backlog, keeps it failing after the trigger ends.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.