Large-scale messaging
On this page 9 topics
Overview
Topics
- Partition strategies
- consumer lag
- rebalancing
- backpressure
Learn first
3.5 h of practice (estimate)
Learning outcomes
- Choose partition counts and keys for throughput, ordering, and future growth.
- Monitor and act on consumer lag with clear thresholds.
- Minimize rebalancing impact (cooperative rebalancing, static membership).
- Apply backpressure end to end so producers cannot overwhelm consumers.
- Plan Kafka cluster capacity, retention, and multi-tenant topic governance.
Lessons
Lab and exercises
Lab
SRB-006-LAB1 · Partner feed at scale
Outcomes: 1, 2, 3, 4, 5 · Time: ~3.5 h
- Estimate volume (50x Shop orders) and growth for three years.
- Design topics, keys, partition counts, and consumer groups.
- Define lag alerts, action plans, and a backlog-recovery plan.
- Run a load test that simulates a backlog and measure catch-up time.
Acceptance: load test proves catch-up time within target; lag alerts have action plans.
Exercises
| ID | Outcome | Task |
|---|---|---|
| SRB-006-EX01 | 1 | Choose partition count for 20,000 msg/s with 2,000 msg/s per consumer and 3x growth. |
| SRB-006-EX02 | 1 | Pick the key for shipment events and justify. |
| SRB-006-EX03 | 2 | Write two lag alerts with thresholds and first actions. |
| SRB-006-EX04 | 3 | A rolling deploy causes repeated stop-the-world rebalances. Propose fixes. |
| SRB-006-EX05 | 4 | Design backpressure for ingest from partner HTTP to Kafka to DB. |
| SRB-006-EX06 | 5 | Compute storage for 5,000 msg/s x 1 KB, 7 days, replication 3. |
Show solutions
- EX01 At least 10 now, 30 for growth; add catch-up headroom, e.g. 36 to 48.
- EX02 shipmentId keeps per-shipment order; avoid carrierId which creates hot keys.
- EX03 Warning: time-lag 2 min sustained; check handler latency and errors. Page: 10 min or rising; scale consumers, check downstream.
- EX04 Cooperative assignor, static membership, longer session timeout, tune max poll interval.
- EX05 Rate limit at the ingest API, bounded in-flight sends, pause consumer on DB slowness, shed or queue policy at the edge.
- EX06 5,000 KB/s x 604,800 s about 3 TB x 3 = about 9 TB; check compression and verify with real data.
Workplace evidence: a messaging scale problem you diagnosed or prevented.
Quiz
Outcomes: O1 partitions · O2 lag · O3 rebalancing · O4 backpressure · O5 capacity and governance.
A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.
Practical part
Design review of the lab's feed design with load test results. Pass: 70% on the quiz and catch-up time within target.
Common mistakes
Partition counts chosen for today; lag alerts without action plans.
Assessment
Design review.
Workplace evidence
A messaging scale problem you diagnosed or prevented.
Career relevance
Platform strategy (STF-004).