عرض عملي: عاصفة إعادة محاولات لا تتوقف من تلقاء نفسها

خطوة بخطوة

1. Start from the symptoms: orders failing and database CPU high, although the original slowdown ended 20 minutes ago.
2. Form hypotheses (the database is still slow, a deploy broke it, a retry loop sustains load) and test each with request-rate and attempt metrics.
3. Find retries at three layers multiplying traffic about 27 times.
4. Planned mistake: restart the order service, and the storm resumes within a minute. Fix: shed load at the edge and disable layered retries first, then restore gradually.
5. Break the loop: enable load shedding, reduce retries to the outer layer only, ramp traffic back.
6. Write the follow-ups: retry budgets, jitter, shrinking timeouts, admission control.

اختبر فهمك