انتقل إلى المحتوى
أمين أبوزيد
Englishالعربية
SRC-015

استكشاف الأخطاء المتقدم

  • 8 درس
  • 4.5 ساعة تدريب (تقدير)
ابدأ الوحدة
في هذه الصفحة 9 موضوعات

نص الدروس والتمارين باللغة الإنجليزية، بينما تبقى واجهة الموقع وعناوين الدروس بالعربية.

نظرة عامة

الموضوعات

  • تجويع الخيوط (Thread starvation)
  • استنزاف الاتصالات
  • الجمود (deadlocks)
  • الأعطال المتتالية (cascading failures)

تعلّم أولًا

4.5 ساعة تدريب (تقدير)

مخرجات التعلم

  1. Diagnose thread starvation, connection exhaustion, and deadlocks across services from telemetry and dumps.
  2. Recognize cascading-failure and metastable-failure patterns (retry storms, queue buildup) and break them.
  3. Form and test hypotheses systematically under time pressure.
  4. Find causes that span layers (JVM, database, network, Kubernetes).

الدروس

  1. 01استكشاف الأخطاء كاختبار للفرضيات
  2. 02تجويع الخيوط
  3. 03استنزاف الاتصالات
  4. 04الجمود الموزّع وقوافل الأقفال (Lock convoys)
  5. 05الأعطال المتتالية والأعطال المستقرة ذاتيًا (Metastable)
  6. 06تحقيقات عبر الطبقات
  7. 07عرض عملي: عاصفة إعادة محاولات لا تتوقف من تلقاء نفسها
  8. 08عرض عملي: زمن استجابة سببه خنق CPU في Kubernetes

المختبر والتمارين

Lab

SRC-015-LAB1 · Mystery outages

Outcomes: 1–4 · Time: ~4.5 h Three timed scenarios on Shop Platform with layered causes (for example thread starvation hiding a connection leak, a retry storm, CPU throttling). For each, keep an investigation log, find the root cause, and propose the fix. Acceptance: root cause, evidence chain, and fix for each within the time box.

Exercises (assessment: live troubleshooting simulation, a graduation gate component)

IDOutcomeTask
SRC-015-EX011Given two thread dumps ten seconds apart showing 190 threads in a payment client read, state the diagnosis, evidence, and fixes.
SRC-015-EX021Compute pool sizing for 30 pods against a database limit of 400 connections, and explain the failure at 10× scale.
SRC-015-EX031Two transactions deadlock updating stock rows. Show how to confirm and fix.
SRC-015-EX042Describe the retry multiplication across gateway, orders, and payment client with 3 attempts each and propose a retry budget.
SRC-015-EX053Write an investigation log for the scenario "p99 spikes, CPU low, DB healthy" with at least three hypotheses and tests.
SRC-015-EX064Case: latency rose after moving to a smaller memory limit. List layers to check and the signal for each.
أظهر الحلول
  • EX01 Thread starvation due to a slow dependency without a timeout. Evidence: same stack in both dumps, pool active at max, low CPU. Fixes: timeouts, bulkhead, load shedding, circuit breaker.
  • EX02 Pool size multiplied by pod count must stay under the limit with headroom, for example 10 per pod gives 300. Scaling pods 10× multiplies connections beyond the limit, so use a pooler or cap pool sizes and shrink transactions.
  • EX03 Confirm from the database deadlock log or lock views; fix by locking rows in sorted ID order and keeping transactions short.
  • EX04 3 × 3 × 3 = 27 calls to the payment service per user request. Retry at one layer only, limit retries to a fraction of traffic (budget), backoff with jitter, shrink timeouts down the chain.
  • EX05 Hypotheses: GC pauses (GC logs), CPU throttling (cgroup metrics), network or DNS delay (hop latencies). Run the cheapest test first; log result and next step.
  • EX06 JVM (GC frequency, heap pressure), container (OOM kills, memory limit), OS (swap, page cache), database (connection churn), network. Compare before and after the change.
  • Outcome 2 is also covered by the lab and Q05-Q06; outcome 3 by Q01 and the investigation logs.

Workplace evidence: a hard production investigation you led, written up.

الاختبار

Outcomes: O1 starvation, exhaustion, deadlocks · O2 cascading and metastable failure · O3 hypothesis testing · O4 cross-layer causes.

اختبار ذاتي لك: لا توجد درجات، ولا يُحفظ أو يُرسل أي شيء. اختر إجابة ثم تحقق منها، ويمكنك المحاولة مرة أخرى كما تشاء.

  1. 1Before restarting a misbehaving service you should:O3
  2. 2Low CPU with rising latency most suggests:O1
  3. 3Two thread dumps showing the same threads in the same call indicate:O1
  4. 4Connection exhaustion can be caused by:O1
  5. 5A metastable failure:O2
  6. 6Three layers each retrying 3 times can multiply calls by:O2
  7. 7To break a retry storm you should first:O2
  8. 8The standard remedy for lock-order deadlocks is:O1
  9. 9Latency spikes with low average CPU in a container may indicate:O4
  10. 10Which is the best use of a hypothesis log?O3

الجزء العملي

Live troubleshooting simulation (SRC-015 assessment, graduation gate component) plus the three SRC-015-LAB1 scenarios with logs. Pass: at least 8/10 on the quiz, and each scenario solved with an evidence chain within its time box.

أخطاء شائعة

Stopping at the first plausible cause; restarting away the evidence.

التقييم

Live troubleshooting simulation (graduation gate component).

دليل من بيئة العمل

A hard production investigation you led, written up.

الصلة بالمسار الوظيفي

Incident leadership (SRC-019) and systemic risk (STF-005).

اكتب للبحث في المستويات والمراحل والوحدات والدروس.