Skip to content
Ameen Abozaid
Englishالعربية
MID-032

Distributed coordination

  • 5 Lessons
  • 2 h practice (estimate)
Start module
On this page 8 topics

Overview

Topics

  • Leases
  • leader election concepts
  • fencing tokens

Learn first

2 h of practice (estimate)

Learning outcomes

  1. Explain leases and leader election and when a system needs a single active worker.
  2. Explain why a lock held via an expired lease is unsafe and how fencing tokens prevent damage.
  3. Implement a single-runner scheduled job across instances (database-based lock, e.g., ShedLock).

Lessons

  1. 01When only one instance may act
  2. 02Leases and leader election
  3. 03Distributed locks and their limits
  4. 04Fencing tokens
  5. 05Demo: a nightly job running once across three replicas

Lab and exercises

Lab

MID-032-LAB1 · Single runner

Outcomes: 1–3 · Time: ~2 h Make the nightly reconciliation job in orders run once across three replicas using a database lock (ShedLock or an equivalent lease table). Add a fencing token: each lock grant increments a number that the job writes with its results, and the update rejects lower tokens.

Acceptance: a test that pauses the lock holder past its lease, lets another instance run, then resumes the first does not double-apply any change.

Exercises (assessment: quiz)

IDOutcomeTask
MID-032-EX011List three Shop jobs and say which truly need a single active worker and which can use another design.
MID-032-EX021Explain leader election and what must happen when the leader dies.
MID-032-EX032Draw the timeline of an unsafe lease: holder pause, expiry, takeover, double write.
MID-032-EX042Write the SQL update that applies a fencing token.
MID-032-EX053Configure lockAtMostFor and lockAtLeastFor for a job that normally takes 3 minutes and at worst 20.
MID-032-EX062Review a design that "uses Redis SET NX for exactly-once payouts" and recommend changes.
Show solutions
  • EX01 Reconciliation: one runner (lock + fencing). Outbox relay: SKIP LOCKED. Email digest: lock or unique "sent for date" record. Cleanup of old rows: idempotent, can run anywhere.
  • EX02 Instances compete for a lease; the winner acts and renews it; if it dies the lease expires and another wins. The old leader must stop on failed renewal.
  • EX03 A acquires; A stalls; lease expires; B acquires and writes; A resumes and writes: both applied.
  • EX04 UPDATE state SET last_token = :t, ... WHERE id = :id AND last_token <= :t and treat 0 rows as stale.
  • EX05 lockAtMostFor of about 30 minutes (above worst case, still releases after a crash); lockAtLeastFor of 1 to 5 minutes to prevent immediate re-run through clock skew.
  • EX06 Redis locks cannot guarantee exclusion under pauses or failover. Make payouts idempotent (unique payout key), add fencing in the payout table, and use the lock only to reduce duplicate work.

Outcome 3 is mainly assessed by the lab and quiz Q09 and Q10.

Quiz

Outcomes: O1 leases and leaders · O2 lock safety and fencing · O3 single-runner jobs.

A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.

  1. 1Three replicas each run a @Scheduled job. By default the job runs:O1
  2. 2A lease differs from a plain lock because it:O1
  3. 3Which design avoids needing a lock for a task queue?O1
  4. 4Why is an expired-lease holder dangerous?O2
  5. 5A fencing token is:O2
  6. 6Where must a fencing check live?O2
  7. 7A Redis SET NX PX lock alone is:O2
  8. 8A write with token 7 arrives after token 8 was accepted. The resource should:O2
  9. 9ShedLock is mainly used to:O3
  10. 10If lockAtMostFor is shorter than the job's runtime:O3

Practical part

Six exercises MID-032-EX01 to EX06 plus MID-032-LAB1. Pass: at least five exercises, 8 of 10 quiz answers, and a paused-instance test with no double-applied change.

Common mistakes

Trusting a Redis lock for correctness; no fencing.

Assessment

Quiz.

Career relevance

Consensus concepts (SRB-003).

Type to search levels, phases, modules and lessons.