Path 7 · lesson 2 of 8
Change data capture with Debezium
Log-based CDC reads the database's write-ahead log and turns every committed row change into a Kafka record. How Debezium does it on Kafka Connect, and what to watch in production.
What you will learn
- Explain how log-based CDC differs from polling a table, and why it sees every committed change in commit order
- Describe how a Debezium PostgreSQL connector runs inside Kafka Connect: replication slot, snapshot, streaming, stored offsets
- Read a Debezium change event (before, after, source, op) and handle deletes and tombstones in a consumer
- Configure the outbox event router so an outbox table becomes clean business events on a topic per aggregate type
- Recognise and prevent the main production risks: WAL retained by an idle slot, schema changes, duplicates after a restart
- Choose between a polling publisher and CDC for a given service
Before you start
The concept
Every relational database already keeps an ordered record of each committed change: PostgreSQL's write-ahead log (WAL), MySQL's binlog. The database writes it for crash recovery and replication. Log-based change data capture (CDC) reads that log as a client and emits one event per inserted, updated or deleted row. The application does nothing extra: if a row changed and the transaction committed, the change is in the log.
Debezium is an open-source set of CDC connectors. Most commonly it runs as Kafka Connect source connectors: Kafka Connect is the framework, shipped with Apache Kafka, that runs connectors in worker processes, stores their configuration and progress in Kafka topics, and restarts their tasks on another worker when one fails. Debezium is not part of Apache Kafka. It is a separate project that you add to a Connect cluster as a plug-in.
- The PostgreSQL connector reads a logical replication stream through a replication slot, using the
pgoutputplug-in that PostgreSQL itself uses for logical replication. - By default each captured table gets its own topic, named
topicPrefix.schemaName.tableName, for exampleshop.public.orders. - Each record's key is the row's primary key, and its value is an envelope with the row state
beforeandafterthe change, the operationop, andsourcemetadata such as the transaction id and log position. - Connect records how far the connector has read (its offsets) in a Kafka topic, so a restarted connector continues where it stopped.
For the order system, CDC has two uses. It can be the relay for the outbox: Debezium captures inserts into outbox_event, and its outbox event router turns each one into a business event on orders.events. It can also publish raw row changes, for example every change to the stock_level table, for consumers that maintain a cache, a search index or a reporting copy.
Polling publisher
- A scheduled query in your own service: nothing new to deploy
- Works on any database with transactions
- Latency is the poll interval, and each poll costs a query even when idle
- Needs a published flag, an index and a cleanup job
- You implement ordering, single-active relay and retries yourself
- Cannot see deletes or intermediate updates of ordinary tables
Log-based CDC with Debezium
- Needs a Kafka Connect cluster, the connector and database replication privileges
- Needs a supported database with logical decoding or binlog access enabled
- Low latency, with no queries against the tables while streaming
- Outbox rows can be deleted in the inserting transaction: the log still has the insert
- Changes arrive in commit order, read by a single task
- A stopped connector makes the database retain WAL: a new operational risk
Why it exists
Before log-based CDC, there were three ways to get data out of a database and into other systems, and each loses something.
| Approach | How it works | What it misses |
|---|---|---|
| Dual write from the application | The service writes to the database and also to Kafka | Atomicity: one of the writes can fail alone (see the outbox lesson). Changes made by scripts, migrations or other services are never published. |
| Polling with a timestamp or version column | SELECT ... WHERE updated_at > :last on a schedule | Deletes (the row is gone), intermediate states between two polls, and rows whose transaction committed late with an older timestamp. It also loads the primary with queries. |
| Triggers writing to an audit table | A trigger copies each change to another table, which is then polled | Nothing in principle, but every write pays for the trigger inside its transaction, and the audit table still has to be polled and cleaned. |
The transaction log has none of these gaps. It contains every committed change, including deletes, in the order the database applied them, whoever made the change. Reading it adds no statements to the application's transactions. The cost is a dependency on database internals: log retention, replication privileges and a format that differs per database. Debezium's job is to hide that behind one event format.
For the outbox specifically, the polling relay in the previous lesson is correct but has limits: latency tied to the poll interval, a query load that grows with the poll rate, and code you own for ordering and failover. Replacing it with CDC removes the flag, the index, the lock and the polling loop. The Order Service keeps only the insert into outbox_event.
How it works inside
Slot, snapshot, streaming
- Prerequisites. PostgreSQL runs with
wal_level=logical. The connector's database user has theREPLICATIONprivilege, and withpgoutputa publication lists the tables whose changes are published. - Replication slot. On first start the connector creates a logical replication slot (default name
debezium). A slot is the server's memory of one consumer: PostgreSQL keeps every WAL segment that the slot's consumer has not yet confirmed. - Snapshot. PostgreSQL normally removes old WAL, so the log does not hold the full history. With the default
snapshot.mode=initial, a connector with no stored offsets first reads the current contents of the captured tables in a consistent snapshot and emits them as events withopset tor(read). - Streaming. The connector then streams changes from the exact log position of the snapshot, so nothing committed during the snapshot is missed. Each change becomes a record sent to Kafka through Connect's producer.
- Offsets. Connect periodically writes the connector's position (the log sequence number, LSN) to its offsets topic. After Kafka has the records, the connector confirms the position to PostgreSQL, which can then discard older WAL.
The PostgreSQL connector always uses a single task, whatever tasks.max says: one reader of one ordered log. That is what preserves commit order, and it also means throughput is bounded by one stream.
The change event, in outline
{ "before": { "sku": "SKU-42", "available": 7 }, "after": { "sku": "SKU-42", "available": 6 }, "source": { "connector": "postgresql", "db": "inventorydb", "schema": "public", "table": "stock_level", "txId": 584, "lsn": 24064704, "snapshot": false }, "op": "u", "ts_ms": 1556890294484}Why it is written this way
opisc(create),u(update),d(delete),r(read during a snapshot) ort(truncate). The record key is the primary key, here thesku.- This example assumes
REPLICA IDENTITY FULLon the table, which puts the whole old row inbeforeat the cost of more WAL. With PostgreSQL's default setting only previous values of primary key columns are available, and for an update only if the key itself changed, so do not build logic onbeforewithout checking the setting. source.lsnandsource.txIdidentify the change in the log. A consumer can store the last LSN it applied to detect duplicates.- A delete is an event with
opdandafternull. By default (tombstones.on.delete=true) it is followed by a second record with the same key and a null value, so a compacted topic can drop the key.
The outbox event router
Consumers of business events should not have to unpack a row-change envelope of an outbox table. The outbox event router is a single message transformation (SMT) that runs inside the connector and rewrites each captured insert:
| Outbox column (default name) | Becomes |
|---|---|
aggregatetype | Part of the topic name. The default pattern is outbox.event. followed by the value. This lesson changes it to the value followed by .events |
aggregateid | The record key, so all events of one aggregate share a partition |
id | A header named id: the event id consumers deduplicate on |
payload | The record value, and nothing else unless you add fields |
type and other columns | Not emitted by default. table.fields.additional.placement can add them as headers or envelope fields |
The router expects an outbox table to behave as a queue: inserts only. It drops delete events for the table automatically and, by default, logs a warning for updates. That is why the Order Service can insert the outbox row and delete it in the same transaction: the log contains both, and only the insert becomes an event.
What happens when something fails
| Crash point | Resulting state | Recovery |
|---|---|---|
| Order Service before its commit | Nothing in the WAL that will be decoded as committed | No event. The transaction never committed, so the connector has no change to publish for it. |
| Connect worker crashes after sending records, before the offset flush | Records are in Kafka, stored offset is older | Another worker restarts the task from the stored offset and emits those changes again. Consumers deduplicate. |
| Connect worker stopped gracefully | Offsets recorded, slot remains | The task moves to another worker and continues where it stopped. |
| Connector stopped or failed for hours | The slot holds its position. PostgreSQL keeps all WAL since then | On restart the connector catches up. If the disk fills first, the database stops accepting writes. |
| Kafka unavailable | Connect cannot produce. The slot position does not advance | The connector retries and resumes when brokers return. WAL accumulates meanwhile. |
| Slot dropped, or invalidated by a WAL size limit | The stored offset points at WAL that no longer exists | Changes in the gap are lost to the stream. A new snapshot is required to resynchronise. |
| Database failover to a standby | The slot may not exist on the new primary | Follow the connector's documented failover steps: recreate the slot before writes resume, and re-snapshot if in doubt. |
Ordering
The connector reads changes in commit order, and all changes to one row carry the same key, so they land in one partition in that order. Across different keys, and across different tables (which are different topics), Kafka gives no relative order. A transaction that touched three tables arrives as independent records on three topics. Consumers that need the whole transaction should be given one outbox event instead of being asked to reassemble it.
Architecture
Drawing diagram…
Flowchart: the Order Service writes orders and outbox rows in one PostgreSQL transaction. Both tables feed the write-ahead log, which a replication slot exposes as a logical replication stream to the Debezium connector running in a Kafka Connect worker. The outbox event router transformation produces records to the orders.events topic, the connector stores its log position in the connect-offsets topic and confirms it back to the slot. Payment and Inventory consume the topic.
flowchart LR
OS["Order Service"] -->|"one transaction"| PG
subgraph PG["PostgreSQL, order database"]
OT["orders"]
OB["outbox_event"]
WAL["Write-ahead log"]
SLOT["Replication slot order_outbox"]
end
OT -.-> WAL
OB -.-> WAL
WAL --> SLOT
subgraph KC["Kafka Connect worker"]
DBZ["Debezium PostgreSQL connector"]
SMT["Outbox event router"]
end
SLOT -->|"logical replication stream"| DBZ
DBZ --> SMT
SMT -->|"key orderId, header id"| T["Topic orders.events"]
DBZ -->|"last LSN read"| OFF["Topic connect-offsets"]
DBZ -.->|"confirm flushed LSN"| SLOT
T --> PS["Payment Service"]
T --> IS["Inventory Service"]- The application's only responsibility is the insert into
outbox_eventin its transaction. Everything from the WAL onwards is infrastructure. - The connector and the transformation run in the same Connect task. The transformation changes the topic, key, headers and value before the record is produced.
- Progress is kept in two places: the offsets topic tells the connector where to resume, and the confirmed position on the slot tells PostgreSQL which WAL it may delete. A connector that is not running confirms nothing.
- Consumers see an ordinary Kafka topic. They do not know or care that the producer was a CDC connector.
Message flow
Drawing diagram…
Sequence diagram: the Order Service inserts an order and an outbox row, deletes the outbox row and commits. PostgreSQL streams the decoded changes to the Debezium connector, whose router publishes the insert to orders.events. The worker crashes before flushing its offset, restarts from an older log position, publishes the same event again, stores the new offset and confirms it to PostgreSQL. The Inventory Service processes the first copy and skips the second.
sequenceDiagram participant O as Order Service participant P as PostgreSQL participant D as Debezium connector participant K as Kafka participant I as Inventory Service O->>P: BEGIN O->>P: INSERT orders row O->>P: INSERT outbox_event row e1 O->>P: DELETE outbox_event row e1 O->>P: COMMIT, changes written to WAL P-->>D: decoded insert and delete at LSN 100 D->>D: router keeps the insert, drops the delete D->>K: orders.events key o-1001 header id e1 K-->>D: ack Note over D: worker crashes before the offset flush D->>K: read connect-offsets after restart K-->>D: last stored LSN 90 D->>P: start replication from LSN 90 P-->>D: insert at LSN 100 again D->>K: orders.events key o-1001 header id e1 D->>K: store offset LSN 100 D->>P: confirm flushed LSN 100 K->>I: e1, first copy, reserve stock K->>I: e1, second copy, skipped by event id
- The outbox row never needs to be visible in the table. Its insert is in the log, and that is what the connector reads.
- Offsets are flushed periodically (the Connect worker default is every 60 seconds), not per record. Everything sent since the last flush can be sent again after a crash.
- The connector confirms a position to PostgreSQL only after it is safely recorded. Until then the WAL from that position onwards is retained.
In a real system
A typical setup for the order system runs a Kafka Connect cluster of three workers in distributed mode. Two connectors are registered. order-outbox-connector captures only public.outbox_event in the order database and routes to orders.events (12 partitions, key = order id), read by the payment-service, inventory-service and notification-service groups. inventory-stock-connector captures public.stock_level in the inventory database and writes raw change events to inventory.public.stock_level (key = sku), read by a service that maintains the product page's availability cache. Each connector has its own replication slot.
Invalidate from the change log
Simulation, not a real clusterThe animated lesson shows the second use: row changes from the log driving cache invalidation, with the ordering and lag effects visible.
Step 1 · The pattern
A copy with a timer
Readers ask the cache first. When the TTL runs out the key is dropped, the next reader fetches the row from the database and stores it again. Nothing is writing, so every copy is correct.
- current data
- request
When to use it
- As the outbox relay when polling is too slow, too heavy on the database, or you have many services and want one standard relay instead of a hand-written one in each.
- To feed derived stores (cache, search index, warehouse) from an existing table without changing the application that writes it.
- To capture changes made outside the application as well: batch jobs, migrations, manual fixes.
- During a migration, to keep a new system's copy of the data in step with the old one.
When not to use it
- Nobody will operate Kafka Connect and watch the replication slot. A neglected slot can fill the database's disk, which is worse than a late event.
- The database is a managed service or version where logical decoding cannot be enabled, or where you cannot get replication privileges.
- One service with modest volume already has a working polling relay. CDC adds infrastructure without changing the guarantee.
- As the contract between teams, when it would expose table structure. Publish outbox events instead.
Java and Spring Boot
A Debezium 3.7 PostgreSQL connector that relays the order outbox, registered with Kafka Connect's REST API (POST /connectors), then two Spring consumers: one for the routed business events and one for raw row changes. Option names and defaults were checked against the Debezium stable documentation. Confirm them for the version you deploy.
{ "name": "order-outbox-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "tasks.max": "1", "database.hostname": "orders-db", "database.port": "5432", "database.user": "debezium", "database.password": "${file:/opt/connect/secrets/orders-db.properties:password}", "database.dbname": "orderdb", "topic.prefix": "orders-cdc", "plugin.name": "pgoutput", "slot.name": "order_outbox", "publication.name": "order_outbox_pub", "publication.autocreate.mode": "filtered", "table.include.list": "public.outbox_event", "tombstones.on.delete": "false", "heartbeat.interval.ms": "10000", "transforms": "outbox", "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter", "transforms.outbox.route.topic.replacement": "${routedByValue}.events", "transforms.outbox.table.fields.additional.placement": "type:header:eventType", "transforms.outbox.table.expand.json.payload": "true", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false" }}Why it is written this way
plugin.namemust be set: the documented default isdecoderbufs, which needs an extra server plug-in.pgoutputis built into PostgreSQL.slot.namemust be unique per connector, and may contain only lower-case letters, digits and underscores. The default namedebeziumcollides as soon as a second connector appears.publication.autocreate.mode=filteredcreates a publication for the included tables only. The defaultall_tablespublishes every table in the database.table.include.listnames only the outbox table. The Debezium documentation says a connector that applies the outbox router should capture outbox tables only.route.topic.replacementturnsaggregatetype = 'orders'into the topicorders.events.type:header:eventTypecopies thetypecolumn into a header so consumers can route without parsing the value.StringConverterwrites the key as the plain order id.table.expand.json.payloadwith the JSON converter and schemas disabled writes the payload as a JSON object, not as an escaped string.tombstones.on.delete=false: deleting an outbox row must not produce a null-value record for that aggregate id. The password uses a Connect config provider. Never put it inline.
-- Order database, prepared for log-based CDC.-- Server setting (postgresql.conf, needs a restart): wal_level = logicalCREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD '...';GRANT SELECT ON public.outbox_event TO debezium;-- Same columns as the polling version. published_at and its index are no longer needed.CREATE TABLE outbox_event ( id uuid PRIMARY KEY, aggregatetype varchar(255) NOT NULL, aggregateid varchar(255) NOT NULL, type varchar(255) NOT NULL, payload jsonb NOT NULL);-- The business transaction. The row is deleted before the commit:-- the WAL contains the INSERT, and that is what Debezium reads.-- BEGIN;-- INSERT INTO orders (...) VALUES (...);-- INSERT INTO outbox_event (id, aggregatetype, aggregateid, type, payload) VALUES (...);-- DELETE FROM outbox_event WHERE id = ...;-- COMMIT;-- Watch the slot: how much WAL is the database keeping for this connector?SELECT slot_name, active, confirmed_flush_lsn, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_walFROM pg_replication_slots;Why it is written this way
- The Order Service's code changes by one statement compared with the polling version: the delete. The relay class, its lock and its cleanup job are removed.
- If you would rather keep rows for inspection, skip the delete and remove old rows with a scheduled job. The router ignores those delete events too.
- The monitoring query is the most important line in the file.
retained_walshould stay small andactiveshould be true. Alert on both. - Creating the publication for specific tables may need table ownership or extra privileges, depending on the PostgreSQL version. The Debezium connector documentation lists the exact grants.
package com.example.inventory;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;/** Consumes business events produced by the outbox event router. */@Componentpublic class OrderEventsListener { public record OrderCreated(String orderId, String customerId, String sku, int quantity, long totalCents) {} private final JdbcTemplate jdbc; private final ObjectMapper json; public OrderEventsListener(JdbcTemplate jdbc, ObjectMapper json) { this.jdbc = jdbc; this.json = json; } @KafkaListener(id = "inventory-orders", topics = "orders.events", groupId = "inventory-service") @Transactional public void onOrderEvent(ConsumerRecord<String, String> record) throws JsonProcessingException { if (!"OrderCreated".equals(header(record, "eventType"))) { return; } // The router put the outbox row's id column into the "id" header. int firstTime = jdbc.update(""" INSERT INTO processed_event (consumer, event_id) VALUES ('inventory-service', ?::uuid) ON CONFLICT DO NOTHING""", header(record, "id")); if (firstTime == 0) { return; // the connector replayed this change after a restart } OrderCreated event = json.readValue(record.value(), OrderCreated.class); jdbc.update(""" INSERT INTO stock_request (order_id, sku, quantity, status) VALUES (?, ?, ?, 'RECEIVED')""", event.orderId(), event.sku(), event.quantity()); } private static String header(ConsumerRecord<String, String> record, String name) { Header h = record.headers().lastHeader(name); if (h == null) { throw new IllegalArgumentException("Missing header " + name); } return new String(h.value(), StandardCharsets.UTF_8); }}Why it is written this way
- This is the same consumer you would write for the polling relay. Replacing the relay with CDC does not change the event contract: topic, key,
idandeventTypeheaders, JSON value. - Header values are bytes. Connect's default header converter writes strings as UTF-8, which is what the helper decodes.
- Deduplication by event id is still required. The connector can replay everything since its last offset flush.
package com.example.catalog;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.JsonNode;import com.fasterxml.jackson.databind.ObjectMapper;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;/** Consumes raw Debezium change events for one table and keeps a local copy. */@Componentpublic class StockLevelChangeListener { private final JdbcTemplate jdbc; private final ObjectMapper json; public StockLevelChangeListener(JdbcTemplate jdbc, ObjectMapper json) { this.jdbc = jdbc; this.json = json; } @KafkaListener(id = "catalog-stock", topics = "inventory.public.stock_level", groupId = "catalog-service") public void onChange(ConsumerRecord<String, String> record) throws JsonProcessingException { if (record.value() == null) { return; // tombstone that follows a delete: nothing more to do } JsonNode envelope = json.readTree(record.value()); String op = envelope.path("op").asText(); long lsn = envelope.path("source").path("lsn").asLong(); switch (op) { case "c", "u", "r" -> { JsonNode after = envelope.path("after"); // Upsert, and ignore anything older than what is already stored. jdbc.update(""" INSERT INTO availability (sku, available, source_lsn) VALUES (?, ?, ?) ON CONFLICT (sku) DO UPDATE SET available = EXCLUDED.available, source_lsn = EXCLUDED.source_lsn WHERE availability.source_lsn < EXCLUDED.source_lsn""", after.path("sku").asText(), after.path("available").asInt(), lsn); } case "d" -> jdbc.update( "DELETE FROM availability WHERE sku = ?", envelope.path("before").path("sku").asText()); default -> { /* truncate and message events are not used here */ } } }}Why it is written this way
- The value is the envelope shown earlier, because the stock connector uses the JSON converter with
schemas.enable=false. With schemas enabled, the envelope sits inside apayloadfield next to aschemafield. r(snapshot read) is handled like a create. After a re-snapshot every row arrives again, and the upsert makes that harmless.- The write is idempotent by construction: an upsert guarded by the source LSN. A replayed or older change does not overwrite newer state, so no processed-event table is needed.
- A null value is the tombstone Debezium emits after a delete by default. A listener that takes the payload as a required parameter would fail on it, so this one takes the whole
ConsumerRecord. - The delete reads the key column from
before. With the default replica identity a delete event carries the previous values of the primary key columns, provided the table has a primary key.
Configuration
Connector options and defaults from the Debezium 3.7 PostgreSQL connector and outbox event router documentation, plus the Connect worker and PostgreSQL settings that decide how much is replayed and how much WAL is kept.
plugin.nameDefault decoderbufs
- What it does
- The logical decoding output plug-in the connector reads from. Supported values are
decoderbufsandpgoutput. - When to change it
- Set
pgoutputunless you have installed decoderbufs on the server. It is always present in PostgreSQL, including on managed services.
slot.nameDefault debezium
- What it does
- The logical replication slot the connector creates and reads. The slot stores the connector's position on the database side.
- When to change it
- Give every connector its own name. Two connectors on one slot will steal each other's changes.
snapshot.modeDefault initial
- What it does
- When to snapshot.
initialsnapshots only when no offsets are stored. Other modes includealways,initial_only,no_dataandwhen_needed. - When to change it
- Keep
initialfor tables whose existing rows consumers need.no_dataskips the data snapshot, suitable only when everything of interest is still in the WAL.
publication.autocreate.modeDefault all_tables
- What it does
- Whether and how the connector creates the
pgoutputpublication.all_tablescreates one for every table in the database. - When to change it
- Use
filteredso only the included tables are published, or create the publication yourself and manage it with your migrations.
heartbeat.interval.msDefault 0 (no heartbeats)
- What it does
- How often the connector emits a heartbeat record. Heartbeats let it record progress even when no captured table changes.
- When to change it
- Set it when captured tables change rarely while the rest of the database is busy. Without heartbeats the connector has nothing to confirm and WAL accumulates.
tombstones.on.deleteDefault true
- What it does
- Whether a delete event is followed by a record with the same key and a null value, so a compacted topic can remove the key.
- When to change it
- Keep
truefor table topics that are compacted. Setfalsefor an outbox connector.
transforms.outbox.route.topic.replacementDefault outbox.event.${routedByValue}
- What it does
- The destination topic name.
${routedByValue}is replaced by the value of the routing column (aggregatetypeby default). - When to change it
- Change it to match your topic naming. Create the topics beforehand with the partition count and replication factor you want.
transforms.outbox.table.op.invalid.behaviorDefault warn
- What it does
- What the router does when it sees an UPDATE on the outbox table:
warn,error(both continue) orfatal(stop the connector). - When to change it
- Leave at
warnand never update outbox rows. This is the reason the CDC version has nopublished_atcolumn.
offset.flush.interval.ms (Connect worker)Default 60000 (1 minute)
- What it does
- How often Connect tries to commit source offsets for tasks. It bounds how much is replayed after a worker crash.
- When to change it
- Lower it to reduce duplicates after a crash, at the cost of more writes to the offsets topic. It never removes the need for idempotent consumers.
max_slot_wal_keep_size (PostgreSQL)Default -1 (unlimited)
- What it does
- The maximum WAL that replication slots may retain. Beyond it, PostgreSQL removes WAL the slot still needs and the slot can no longer be used.
- When to change it
- Setting a limit protects the database's disk at the price of the connector: an invalidated slot means a new snapshot. Decide which failure you prefer, and alert long before either.
Hands-on exercise
Trace an outbox row through the log, and price an outage
Follow one event from the transaction to the consumer, find where a duplicate comes from, and work out how long a stopped connector can be tolerated before the database is at risk.
Steps · 0 of 5 done
What you should see
The offsets topic decides the replay, the slot decides WAL retention. Up to about 2,000 events (40 s × 50/s) can be republished after the crash, all with event ids the consumers have already recorded. The disk fills after 30 hours, around 04:00 on Sunday, at which point the order database stops accepting writes. Reasonable alerts: slot active false for more than 5 minutes, and retained_wal above a few GB.
What can go wrong
A stopped connector fills the database disk
- Symptom
- Disk usage on the PostgreSQL primary climbs steadily.
pg_walis tens of gigabytes. Eventually writes fail and checkout is down. - Cause
- A replication slot makes PostgreSQL keep all WAL its consumer has not confirmed. The connector was failed, paused, or deleted without dropping its slot, and with
max_slot_wal_keep_sizeat its default there is no limit. - Fix
- Alert on slot inactivity and retained WAL size. Restart or fix the connector. If a connector is retired, drop its slot. Consider a WAL size limit, knowing that hitting it invalidates the slot.
WAL grows although the connector is running
- Symptom
- The connector is healthy and lag metrics look fine, yet retained WAL keeps growing.
- Cause
- The captured tables change rarely while other tables or other databases on the same server are busy. The connector emits nothing, so no offset is committed and no position is confirmed to the server.
- Fix
- Set
heartbeat.interval.ms. For the multi-database case, follow the connector documentation's guidance on a heartbeat table that produces a real change in the captured database.
Duplicates after a Connect restart
- Symptom
- After a worker crash or a rebalance of the Connect cluster, consumers see a burst of events they have already processed.
- Cause
- Offsets are flushed periodically. The task restarts from the last stored offset and emits everything after it again.
- Fix
- Deduplicate in every consumer: by the
idheader for outbox events, bysource.lsnor an idempotent upsert for raw changes.
Schema changes surprise the consumers
- Symptom
- After a migration, a consumer of a table topic fails to deserialise, or silently reads null for a renamed column.
- Cause
- The envelope's
beforeandaftermirror the table's columns. PostgreSQL logical decoding does not carry DDL, so there is no schema change event to react to: the rows simply start arriving in the new shape. - Fix
- Treat captured tables as a published schema: add columns as nullable, never rename or drop in one step, and use a schema registry with compatibility checks for table topics. Prefer outbox events for contracts between teams.
Updating or reusing outbox rows
- Symptom
- The connector log fills with warnings from the event router, and some expected events never appear.
- Cause
- The application updates outbox rows (for example a leftover
published_atflag). The router treats the outbox as insert-only and does not emit events for updates. - Fix
- Insert only. Remove status columns from the CDC version of the table and delete rows instead of updating them.
Snapshot of a large table at the wrong moment
- Symptom
- Registering a connector, or losing its offsets, floods a topic with millions of
revents and puts read load on the primary for hours. - Cause
- With
snapshot.mode=initiala connector that finds no stored offsets snapshots every captured table. Renaming the connector or changingtopic.prefixmakes it look new. - Fix
- Plan the first snapshot. Keep connector names and prefixes stable, restrict
table.include.list, and read the documentation on incremental snapshots for large tables.
How to troubleshoot
No new events on the topic
Check
curl -s http://connect:8083/connectors/order-outbox-connector/status
- How to read it
- Kafka Connect's REST API reports the connector and task state.
FAILEDon the task comes with a stack trace in thetracefield: fix the cause, thenPOST /connectors/order-outbox-connector/tasks/0/restart. A failed task does not restart by itself.
Database disk usage is climbing
Check
SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal FROM pg_replication_slots;
- How to read it
active = falsemeans no client is connected to the slot: the connector is down. A largeretained_walwithactive = truemeans the connector is connected but not confirming progress, which points to missing heartbeats or a connector that cannot produce to Kafka.
Events arrive late
Check
JMX metric `MilliSecondsBehindSource` on the MBean `debezium.postgres:type=connector-metrics,context=streaming,server=<topic.prefix>`
- How to read it
- It is the delay between a change in the database and the connector processing it. A steadily rising value means the single task cannot keep up or Kafka is slow to accept records. A flat low value means the delay is downstream, in the consumers.
Unsure where a connector will resume
Check
curl -s http://connect:8083/connectors/order-outbox-connector/offsets
- How to read it
- Shows the stored source offset, including the LSN. Compare it with
confirmed_flush_lsnon the slot. If the stored LSN is no longer available on the server, the connector cannot continue from it and needs a snapshot.
Consumers lag behind the CDC topic
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group inventory-service
- How to read it
- Lag here is an ordinary consumer problem, unrelated to the connector. After a snapshot or a catch-up the topic receives a burst, and lag that then shrinks steadily is normal.
Key takeaways
- Log-based CDC reads the database's own transaction log: every committed change, including deletes, in commit order, with no extra statements in application transactions.
- Debezium runs as Kafka Connect source connectors. It is a separate open-source project, not part of Apache Kafka.
- A PostgreSQL connector snapshots first, then streams from a replication slot. Its position is stored in Connect's offsets topic and confirmed back to the slot.
- Delivery is at-least-once: a restart replays everything since the last offset flush, so consumers deduplicate.
- The outbox event router turns outbox inserts into business events:
aggregatetypeto topic,aggregateidto key,idto a header,payloadto the value. - A slot nobody reads keeps WAL forever. Monitor slot activity and retained WAL before anything else.
- Order is per key within a partition. Transactions across tables are not kept together.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
On Monday morning the order database's disk is at 94% and rising. pg_replication_slots shows slot order_outbox with active = false and 310 GB of retained WAL. The connector status shows its task FAILED since Saturday with an authentication error: the database password was rotated on Friday.
What should be done first?
Scenario 2 of 3
A Connect worker is killed by the node's out-of-memory killer. Within a minute the task is running on another worker. The Payment Service's dedupe table then rejects 1,150 OrderCreated events in a few seconds as already processed. The team is worried that the connector is broken.
What is the correct reading?
Scenario 3 of 3
A team of four runs one Spring Boot service with PostgreSQL and about 20 orders per minute. Events must reach other services within a few seconds. There is no Kafka Connect cluster in the company and nobody on call for the database at weekends. They are choosing a relay for their new outbox.
Which relay fits this situation?
References
References: Apache Kafka 4.1 documentation: Kafka Connect user guide (workers, REST API, offsets, exactly-once source support) ↗ · Apache Kafka 4.1: Kafka Connect configuration reference (offset.flush.interval.ms, header.converter) ↗ · Debezium documentation (stable, 3.7): Architecture ↗ · Debezium documentation (stable, 3.7): Connector for PostgreSQL (snapshots, change events, properties, WAL disk space, behaviour when things go wrong) ↗ · Debezium documentation (stable, 3.7): Outbox event router ↗ · Debezium blog: Reliable microservices data exchange with the outbox pattern ↗ · PostgreSQL documentation: replication settings (max_slot_wal_keep_size) ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener annotation ↗ · Chris Richardson, microservices.io: Transaction log tailing pattern ↗ · Report a problem