Demo: a retry storm that won't stop by itself
On This Lesson 3 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
diagnose and break a metastable retry storm; prepares SRC-015-LAB1.
Draft outline details
Step by step
- Start from the symptoms: orders failing and database CPU high, although the original slowdown ended 20 minutes ago.
- Form hypotheses (the database is still slow, a deploy broke it, a retry loop sustains load) and test each with request-rate and attempt metrics.
- Find retries at three layers multiplying traffic about 27 times.
- Planned mistake: restart the order service, and the storm resumes within a minute. Fix: shed load at the edge and disable layered retries first, then restore gradually.
- Break the loop: enable load shedding, reduce retries to the outer layer only, ramp traffic back.
- Write the follow-ups: retry budgets, jitter, shrinking timeouts, admission control.
Check your understanding
Why did the restart not help?
Show answer
Clients' retries immediately recreated the overload.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.