Advanced troubleshooting
On this page 9 topics
Overview
Topics
- Thread starvation
- connection exhaustion
- deadlocks
- cascading failures
Learn first
4.5 h of practice (estimate)
Learning outcomes
- Diagnose thread starvation, connection exhaustion, and deadlocks across services from telemetry and dumps.
- Recognize cascading-failure and metastable-failure patterns (retry storms, queue buildup) and break them.
- Form and test hypotheses systematically under time pressure.
- Find causes that span layers (JVM, database, network, Kubernetes).
Lessons
Lab and exercises
Lab
SRC-015-LAB1 · Mystery outages
Outcomes: 1–4 · Time: ~4.5 h Three timed scenarios on Shop Platform with layered causes (for example thread starvation hiding a connection leak, a retry storm, CPU throttling). For each, keep an investigation log, find the root cause, and propose the fix. Acceptance: root cause, evidence chain, and fix for each within the time box.
Exercises (assessment: live troubleshooting simulation, a graduation gate component)
| ID | Outcome | Task |
|---|---|---|
| SRC-015-EX01 | 1 | Given two thread dumps ten seconds apart showing 190 threads in a payment client read, state the diagnosis, evidence, and fixes. |
| SRC-015-EX02 | 1 | Compute pool sizing for 30 pods against a database limit of 400 connections, and explain the failure at 10× scale. |
| SRC-015-EX03 | 1 | Two transactions deadlock updating stock rows. Show how to confirm and fix. |
| SRC-015-EX04 | 2 | Describe the retry multiplication across gateway, orders, and payment client with 3 attempts each and propose a retry budget. |
| SRC-015-EX05 | 3 | Write an investigation log for the scenario "p99 spikes, CPU low, DB healthy" with at least three hypotheses and tests. |
| SRC-015-EX06 | 4 | Case: latency rose after moving to a smaller memory limit. List layers to check and the signal for each. |
Show solutions
- EX01 Thread starvation due to a slow dependency without a timeout. Evidence: same stack in both dumps, pool active at max, low CPU. Fixes: timeouts, bulkhead, load shedding, circuit breaker.
- EX02 Pool size multiplied by pod count must stay under the limit with headroom, for example 10 per pod gives 300. Scaling pods 10× multiplies connections beyond the limit, so use a pooler or cap pool sizes and shrink transactions.
- EX03 Confirm from the database deadlock log or lock views; fix by locking rows in sorted ID order and keeping transactions short.
- EX04 3 × 3 × 3 = 27 calls to the payment service per user request. Retry at one layer only, limit retries to a fraction of traffic (budget), backoff with jitter, shrink timeouts down the chain.
- EX05 Hypotheses: GC pauses (GC logs), CPU throttling (cgroup metrics), network or DNS delay (hop latencies). Run the cheapest test first; log result and next step.
- EX06 JVM (GC frequency, heap pressure), container (OOM kills, memory limit), OS (swap, page cache), database (connection churn), network. Compare before and after the change.
- Outcome 2 is also covered by the lab and Q05-Q06; outcome 3 by Q01 and the investigation logs.
Workplace evidence: a hard production investigation you led, written up.
Quiz
Outcomes: O1 starvation, exhaustion, deadlocks · O2 cascading and metastable failure · O3 hypothesis testing · O4 cross-layer causes.
A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.
Practical part
Live troubleshooting simulation (SRC-015 assessment, graduation gate component) plus the three SRC-015-LAB1 scenarios with logs. Pass: at least 8/10 on the quiz, and each scenario solved with an evidence chain within its time box.
Common mistakes
Stopping at the first plausible cause; restarting away the evidence.
Assessment
Live troubleshooting simulation (graduation gate component).
Workplace evidence
A hard production investigation you led, written up.
Career relevance
Incident leadership (SRC-019) and systemic risk (STF-005).