Skip to content
Ameen Abozaid
Englishالعربية
MID-029

Reliable messaging

  • 7 Lessons
  • 4 h practice (estimate)
Start module
On this page 8 topics

Overview

Topics

  • Transactional outbox
  • inbox
  • retries
  • deduplication
  • idempotency

Learn first

4 h of practice (estimate)

Learning outcomes

  1. Explain the dual-write problem (database + broker) and its failure modes.
  2. Implement the transactional outbox with a relay (polling or CDC).
  3. Make consumers idempotent with an inbox/processed-message table or natural idempotency.
  4. Design retry policies with backoff and limits, and route permanent failures to a DLQ.

Lessons

  1. 01The dual-write problem
  2. 02Transactional outbox
  3. 03Outbox relays: polling vs CDC
  4. 04Idempotent consumers and the inbox
  5. 05Retries, backoff, and DLQs
  6. 06Demo: losing an event without an outbox
  7. 07Demo: processing the same message twice, safely

Lab and exercises

Lab

MID-029-LAB1 · Reliable order events

Outcomes: 1–4 · Time: ~4 h

  1. In orders, add an outbox table and write an OrderPlaced row in the same transaction as the order.
  2. Build a polling relay (SKIP LOCKED, batch of 100) that publishes to Kafka keyed by order id and marks rows sent.
  3. In inventory and payments, add an inbox table (event id primary key) and apply effects in the same transaction as the inbox insert.
  4. Configure retries with exponential backoff and a limit, and a DLQ for permanent failures.
  5. Write a chaos script that kills each service (orders, relay, inventory, payments, Kafka) at random points during a stream of orders, then restarts them.

Acceptance: after the chaos script, every placed order produced exactly one stock reservation and one payment authorization: no lost and no double-applied events.

Exercises (assessment: failure-injection review)

IDOutcomeTask
MID-029-EX011List the two orderings of "commit" and "publish" and the failure each allows.
MID-029-EX022Write the SQL for an outbox table and the service method that writes order and event atomically.
MID-029-EX032Compare polling and CDC relays in a table: latency, load, infrastructure, ordering.
MID-029-EX043Classify as naturally idempotent or not: set status to SHIPPED; add 5 to stock; insert a row with a fixed id; send an email. Make the non-idempotent ones safe.
MID-029-EX053Implement an inbox check that works under two concurrent deliveries of the same event.
MID-029-EX064Choose retry limits and backoff for a payment-authorization consumer; list two errors that go straight to the DLQ.
MID-029-EX071–4Failure-injection review: for each step of the pipeline, state what happens if the process dies right there.
Show solutions
  • EX01 Commit then publish: crash between loses the event. Publish then commit: crash or rollback publishes an event for state that does not exist.
  • EX02 outbox(id, aggregate_id, type, payload, created_at, sent_at); method annotated @Transactional calls orders.save(o) and outbox.save(...); no broker call inside.
  • EX03 Polling: seconds of latency, extra DB queries, no new infrastructure, ordered by created_at. CDC: very low latency, reads the log, needs connectors and ops, ordered by log position.
  • EX04 SHIPPED and fixed-id insert are idempotent. Add 5 to stock and send email are not: record the event id in an inbox (or keep a per-event ledger row) so the effect runs once; for email, store a "sent" flag keyed by event id.
  • EX05 Insert the event id with a primary key in the same transaction as the effect; the second transaction hits a unique violation and rolls back or skips. INSERT ... ON CONFLICT DO NOTHING and check the row count.
  • EX06 Example: 5 attempts, 500 ms start, factor 2, capped at ~30 s; permanent: malformed JSON, unknown product id. Timeouts and 503s retry.
  • EX07 Table of steps: after save+outbox commit (event still in outbox, relay sends later), after publish before mark-sent (duplicate published, inbox absorbs), after inbox insert before commit (rolled back together, redelivered), after DLQ publish (offset committed).

All four outcomes are also exercised by the quiz and by the lab's chaos criterion.

Quiz

Outcomes: O1 dual write · O2 transactional outbox · O3 idempotent consumers · O4 retries and DLQ.

A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.

  1. 1A service commits an order, then crashes before publishing. The result is:O1
  2. 2Why does publishing inside the DB transaction not solve dual write?O1
  3. 3In the outbox pattern the event row is written:O2
  4. 4The delivery guarantee of an outbox relay is normally:O2
  5. 5FOR UPDATE SKIP LOCKED helps a polling relay by:O2
  6. 6A benefit of CDC over polling:O2
  7. 7Which operation is naturally idempotent?O3
  8. 8Where should processed message ids be stored?O3
  9. 9A malformed payload should:O4
  10. 10Exponential backoff means:O4

Practical part

Seven exercises MID-029-EX01 to EX07 plus the failure-injection review and MID-029-LAB1. Pass: at least six exercises and a chaos run with no lost and no double-applied events.

Common mistakes

Publishing inside the DB transaction; deduplication without a durable record.

Assessment

Failure-injection review.

Career relevance

Required in the MID capstone; CDC (SRB-007).

Type to search levels, phases, modules and lessons.