Skip to content
Ameen Abozaid
Englishالعربية
SRB-007

Data platform integration

  • 7 Lessons
  • 4 h practice (estimate)
Start module
On this page 9 topics

Overview

Topics

  • CDC
  • Debezium
  • stream processing
  • analytics pipelines

Learn first

4 h of practice (estimate)

Learning outcomes

  1. Capture changes from a legacy database with CDC (Debezium) without changing its code.
  2. Distinguish CDC events from domain events and translate between them.
  3. Build stream processing for enrichment and aggregation (Kafka Streams or Flink concepts).
  4. Feed analytics pipelines while respecting data ownership and privacy.

Lessons

  1. 01Integrating systems you can't change
  2. 02Change data capture with Debezium
  3. 03From CDC events to domain events
  4. 04Stream processing concepts
  5. 05Analytics pipelines and data ownership
  6. 06Demo: streaming FulfillCo's stock table without touching its code
  7. 07Demo: filtering personal data out of a CDC stream

Lab and exercises

Lab

SRB-007-LAB1 · Stock sync

Outcomes: 1, 2, 3, 4 · Time: ~4 h

  1. Set up Debezium on a copy of FulfillCo's database with a table include list.
  2. Translate change events into the StockLevelChanged domain event.
  3. Consume it in Shop's inventory and measure end-to-end delay.
  4. Exclude personal data and record replication slot monitoring.

Acceptance: Shop sees stock changes within seconds; FulfillCo code unchanged.

Exercises

IDOutcomeTask
SRB-007-EX011List three constraints in capturing from legacy and the CDC mitigation for each.
SRB-007-EX021Describe monitoring for a PostgreSQL replication slot.
SRB-007-EX032Translate this row change (qty 12 to 9) into a domain event and explain what is unknown.
SRB-007-EX042Review a design that exposes the raw stock_levels topic to all teams. Write the critique.
SRB-007-EX053Design an aggregation of stock per warehouse per minute and state the late-event policy.
SRB-007-EX064Plan pushing stock data to analytics with personal data removed and deletion handled.
Show solutions
  • EX01 No code change: log-based; load: reads WAL; ordering: transaction log order and source positions.
  • EX02 Alert on retained WAL size and slot lag; plan for a stalled connector.
  • EX03 StockLevelChanged with available 9; the reason is unknown unless more context; do not guess.
  • EX04 Couples consumers to the schema, leaks columns, no owner; add a translation layer and a contract.
  • EX05 Tumbling 1-minute windows with a grace period, late events dropped or counted separately.
  • EX06 Source-side filter, data product contract, retention and deletion propagation, lineage.

Workplace evidence: a legacy integration you delivered with CDC or a similar technique.

Quiz

Outcomes: O1 CDC · O2 CDC vs domain events · O3 stream processing · O4 analytics and ownership.

A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.

  1. 1Debezium on PostgreSQL reads:O1
  2. 2An unmonitored replication slot can:O1
  3. 3CDC avoids changing legacy code because it:O1
  4. 4A CDC event describes:O2
  5. 5Public contracts should be:O2
  6. 6Translation from CDC to domain events belongs in:O2
  7. 7Event time differs from processing time because:O3
  8. 8Stateful stream state is recovered from:O3
  9. 9Personal data should be filtered:O4
  10. 10Who owns data given to analytics?O4

Practical part

Integration design review of the lab's stock sync, including privacy controls. Pass: 70% on the quiz and a design with a translation layer and filtered personal data.

Common mistakes

Exposing raw table changes as a public contract; CDC on tables with personal data without filtering.

Assessment

Integration design review.

Workplace evidence

A legacy integration you delivered with CDC or a similar technique.

Career relevance

Integration patterns (ARC-024) and data architecture (ARC-023).

Type to search levels, phases, modules and lessons.