Apache Kafka
On this page 8 topics
Overview
Topics
- Brokers
- topics
- partitions
- consumer groups
- offsets
- ordering
Learn first
4 h of practice (estimate)
Learning outcomes
- Explain brokers, topics, partitions, replicas, and the log model.
- Choose partition keys to preserve the ordering that matters.
- Explain consumer groups, offsets, commits, and rebalancing.
- Produce and consume with Spring for Apache Kafka, with error handling and a DLT.
- Configure producer durability (
acks, idempotent producer).
Lessons
Lab and exercises
Lab
MID-028-LAB1 · Order events on Kafka
Outcomes: 1–5 · Time: ~4 h
- Create
orders(3 partitions).orderspublishesOrderPlacedkeyed by order ID withacks=alland idempotence enabled. inventoryandpaymentsconsume in separate groups with aDefaultErrorHandler, back-off, and a DLT.- Make handlers idempotent by
eventId. - Test: events of one order stay in order; restart a consumer mid-stream and confirm nothing is lost; send a poison record and find it in the DLT.
Acceptance: per-order ordering preserved; consumer restart loses nothing.
Exercises (assessment: partitioning and ordering scenarios; six exercises)
| ID | Outcome | Task |
|---|---|---|
| MID-028-EX01 | 1 | Draw a topic with 4 partitions and replication factor 3 over 3 brokers; mark leaders and followers. |
| MID-028-EX02 | 2 | Pick a key for: order events, per-customer notifications, sensor readings per device. Say what ordering you get. |
| MID-028-EX03 | 3 | A group has 3 consumers and a topic has 2 partitions. Describe the assignment and what happens when one consumer leaves. |
| MID-028-EX04 | 3 | Explain the failure when offsets are committed before processing and the failure when committed after. |
| MID-028-EX05 | 4 | Configure a Spring Kafka error handler that retries 3 times, 1 s apart, then publishes to a DLT. |
| MID-028-EX06 | 5 | Choose acks, min.insync.replicas, and idempotence for payment events and explain the trade-off. |
Show solutions
- EX01 Each partition has one leader and two followers on different brokers; leaders spread across brokers.
- EX02 Order events:
orderId(per-order order). Notifications:customerId. Sensors:deviceId. No ordering across keys. - EX03 Two consumers get one partition each; one idles. If an active one leaves, rebalance gives its partition to the idle one.
- EX04 Commit before: crash loses the record. Commit after: crash causes redelivery, so handlers must be idempotent.
- EX05
new DefaultErrorHandler(new DeadLetterPublishingRecoverer(template), new FixedBackOff(1000L, 3)). - EX06
acks=all, replication factor 3,min.insync.replicas=2,enable.idempotence=true; cost is extra latency and reduced availability if two replicas are down.
Quiz
Outcomes: O1 log model · O2 keys and ordering · O3 groups and offsets · O4 Spring and error handling · O5 producer durability.
A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.
Practical part
Six exercises MID-028-EX01 to EX06 plus the lab. Pass: at least five exercises correct, per-order ordering demonstrated, no loss on consumer restart, and a poison record found in the DLT.
Common mistakes
Random keys where order matters; auto-commit before processing; one partition for everything.
Assessment
Partitioning and ordering scenarios.
Career relevance
Large-scale messaging (SRB-006).