{"canonical_url":"https://engineer.company/glossary/change-data-capture/","content_markdown":"# Change data capture\n\n## What it is\n\nChange data capture, usually shortened to CDC, means reading the changes made in\na database and passing them on to other systems as they happen. Every insert,\nupdate and delete becomes an event that something else can act on: a reporting\ncopy that stays current, a search index, another service that needs to know an\norder was placed. The database where the change happens stays the system of\nrecord, and the other systems follow it.\n\nThere are two broad ways to find the changes. The simpler one is to ask: run a\nquery every few minutes for rows whose \"last updated\" timestamp is newer than\nthe previous run. The other is to read the database's own record of what it\nchanged. PostgreSQL keeps that record in its write-ahead log (WAL), and its\ndocumentation states the rule behind it: changes to data files are written only\nafter the records describing them have been flushed to permanent storage.\nBecause the log already holds every change in the order it happened, reading it\ngives a complete history without querying the tables at all. This second\napproach, log-based CDC, is what the term usually means today.\n\nIn PostgreSQL the mechanism is called logical decoding, which the documentation\ndefines as \"extracting all persistent changes to a database's tables\" into a\nformat that can be read without knowing the database's internals. Three parts\nwork together. The server has to write enough detail into the WAL, which the\n`wal_level` setting controls: its default, `replica`, is enough for copying data\nbetween servers, and `logical` adds what decoding needs. A replication slot\nrepresents one stream of changes from one database, delivered to its reader in\nthe original sequence, and it remembers how far that client has read. An output\nplugin turns the WAL's internal records into whatever format the client wants.\n\nMost teams do not write the client themselves. Debezium supplies Apache Kafka\nConnect with connectors that obtain changes through each source database's CDC\nmechanism. Its PostgreSQL connector reads through `pgoutput`, which its\ndocumentation describes as PostgreSQL's built-in output plugin, or through\n`decoderbufs`, a plugin maintained by the Debezium community. Debezium can also\nrun as a library inside a JVM application, or as Debezium Server, which sends\nchange events to systems such as Amazon Kinesis, Google Cloud Pub/Sub and Apache\nPulsar.\n\n## Why it matters\n\nPolling looks cheaper, and for a small table that changes slowly it often is.\nDebezium's documentation lists what reading the log does that polling cannot. It\ncaptures every change, where a query sees only the state of a row at the moment\nit runs, so a row that changed three times between two runs shows up once. It\ncaptures deletes, where a deleted row is simply missing from the next result. It\nneeds no extra column in the data model, such as a \"Last Updated\" field that\nevery write path has to remember to set. It can carry the old state of a row and\nmetadata such as the ID of the transaction that made the change. And it delivers\nchanges with very little delay without the CPU cost of frequent polling: for\nPostgreSQL and MySQL, Debezium puts that delay in the millisecond range.\n\nThe second reason is agreement between a database and the messages other systems\nreceive. Debezium names dual writes as the other approach it compares itself\nwith: the application saves a change to its database and then sends a message\nabout it. If the process stops between those two steps, the database and the\nmessage stream disagree, and nothing records which one is wrong. CDC removes the\nsecond write. The application commits once, and the message is derived from what\nwas committed. The outbox pattern goes a step further: the application writes\nthe message itself into an outbox table in the same transaction as the change,\nand CDC publishes the rows of that table. Debezium's Outbox Event Router\ndescribes the result as avoiding inconsistencies between a service's own state\nand the events that other services consume.\n\nThe third reason is order. A replication slot preserves the source server's\nsequence when it delivers changes, grouped by the transaction that made them, as\nthe example below shows. A consumer that applies them in that order rebuilds the\nsame state, step by step. For an ETL load this makes incremental loading honest:\na nightly job can take exactly the changes since its last run, deletes included,\ninstead of reloading whole tables or trusting a timestamp column that one code\npath forgot to update.\n\n## What it looks like in practice\n\nThe smallest working example uses PostgreSQL alone. It needs `wal_level =\nlogical` in the server configuration, which takes effect only after a restart,\nand a free replication slot; `max_replication_slots` defaults to 10. The session\nbelow runs as a superuser, as the documentation's own example does, and uses\n`test_decoding`, the output plugin from that example, which prints each change\nas a line of text.\n\n```sql\n-- postgresql.conf: wal_level = logical (takes effect after a restart)\nSELECT * FROM pg_create_logical_replication_slot('shop_cdc', 'test_decoding');\n\nBEGIN;\nINSERT INTO orders (id, status) VALUES (42, 'pending');\nINSERT INTO outbox (id, event) VALUES (7, 'order 42 placed');\nCOMMIT;\n\nSELECT data FROM pg_logical_slot_peek_changes('shop_cdc', NULL, NULL);\n-- BEGIN 10301\n-- table public.orders: INSERT: id[integer]:42 status[text]:'pending'\n-- table public.outbox: INSERT: id[integer]:7 event[text]:'order 42 placed'\n-- COMMIT 10301\n\nSELECT data FROM pg_logical_slot_get_changes('shop_cdc', NULL, NULL);\n-- the same four rows; this call consumes them, so a second call returns none\n\nSELECT pg_drop_replication_slot('shop_cdc');\n```\n\nThe order and its outbox message come out together, inside the transaction that\nwrote them. `pg_logical_slot_peek_changes` reads without moving the slot, so it\nreturns the same rows on the next call, while `pg_logical_slot_get_changes`\nconsumes them. A `CREATE TABLE` would produce only an empty `BEGIN` and\n`COMMIT`, because logical decoding does not carry DDL. The last line matters\nmost: a slot left behind keeps holding on to server resources.\n\nIn production, Debezium plays the reader. On first start, its PostgreSQL\nconnector takes a consistent snapshot of the tables it captures and then streams\nchanges from the point where the snapshot was taken. The default\n`snapshot.mode`, `initial`, takes that snapshot only when no stored offsets\nexist, so a restart resumes from the last recorded position instead. Each event\ncarries `before` and `after` images of the row, a `source` block, a timestamp\nand an `op` code: `c` for create, `u` for update, `d` for delete, `r` for a row\nread during a snapshot and `t` for truncate. After a delete, the connector also\nemits a tombstone, an event with the same key and a null value, so that Kafka's\nlog compaction can remove the earlier messages for that key.\n\nBoth patterns from this article show up in the company's own pages. Between 2022\nand 2024, the founder\n[engineered 600 ETL/ELT pipelines in PL/pgSQL](/portfolio/engineered-600-pl-pgsql-etl-elt-pipelines/)\nacross several PostgreSQL development and production environments, including\nincremental loads using CDC and timestamp-based deltas. The note\n[Selling the last ticket exactly once](/notes/selling-limited-stock-without-overselling/)\ndesigns a flash sale around the outbox: every step records its message in the\noutbox within the same transaction as the change, change data capture publishes\nthe outbox, and because a message can arrive more than once, every consumer is\nidempotent, keyed by the order.\n\n## Where it goes wrong\n\n**Nothing caps what a slot may keep.** A slot holds on to every piece of WAL its\nconsumer has not confirmed, and PostgreSQL's defaults set no ceiling on that.\n`max_slot_wal_keep_size` is `-1`, which lets slots retain an unlimited amount of\nWAL, and `idle_replication_slot_timeout` is `0`, which switches off the\ninvalidation of slots that sit unused. A connector that stopped during a failed\ndeploy can therefore fill the disk long after everyone has forgotten it. Watch\n`pg_replication_slots` for slots that are no longer active, and set\n`max_slot_wal_keep_size` to what the disk can afford, knowing that a consumer\nwhich falls further behind than that may no longer be able to continue.\nRecovering from that means a new slot, and a new slot starts from the present:\nDebezium warns that a dropped and recreated slot can silently skip the events in\nbetween, so take a fresh snapshot afterwards.\n\n**A quiet table holds the log back.** Debezium's documentation describes a\nsubtler version. A slot belongs to one database, and the connector advances its\nconfirmed position only when it sees changes it captures. If the captured tables\nrarely change while other tables, or other databases on the same server, write\nheavily, the position barely moves and the WAL behind it keeps growing. The fix\nis a heartbeat: `heartbeat.interval.ms` makes the connector emit heartbeat\nevents at a set interval, and `heartbeat.action.query` lets it write regularly\nto a small table in the captured database, which has to be part of the\npublication the connector reads, so the slot advances even when nothing else\nhappens there.\n\n**Consumers assume each change arrives once.** A logical slot emits each change\nonce in normal operation, but its position is saved only at checkpoints, so\nafter a crash it can return to an earlier point and send recent changes again.\nPostgreSQL's documentation leaves the consequences to the client. A consumer\nthat adds each amount to a running total counts those changes twice. Make every\nconsumer idempotent: apply each change as an upsert keyed by the row's primary\nkey, or record the IDs of the events already processed, so that a second\ndelivery changes nothing.\n\n**Updates and deletes arrive without the old values.** What an update or delete\nevent says about the previous row depends on the table's replica identity. With\n`REPLICA IDENTITY DEFAULT`, Debezium's events carry at most the previous\nprimary-key values, and the connector produces neither update nor delete events\nwhen the table has no primary key. `REPLICA IDENTITY FULL` puts every previous\ncolumn into `before`, at a price: PostgreSQL's documentation notes that it\nincreases WAL volume, particularly when many `UPDATE` and `DELETE` statements\nrun. Give every captured table a primary key, and switch to `FULL` only on the\ntables whose consumers need the old values.\n\n## Further reading\n\n- [Debezium Features](https://debezium.io/documentation/reference/stable/features.html)\n  — what log-based CDC captures that polling and dual writes miss, plus\n  snapshots, filters and masking.\n- [Debezium connector for PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html)\n  — output plugins, snapshots, the change-event format, replica identity and WAL\n  disk space.\n- [Logical Decoding Concepts](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html)\n  — replication slots, output plugins, and why a slot nobody needs should be\n  dropped.\n- [Logical Decoding Examples](https://www.postgresql.org/docs/current/logicaldecoding-example.html)\n  — creating a slot, peeking at and consuming changes, and dropping the slot\n  again.\n- [Replication (server configuration)](https://www.postgresql.org/docs/current/runtime-config-replication.html)\n  — `max_replication_slots`, `max_slot_wal_keep_size` and\n  `idle_replication_slot_timeout`.\n- [Outbox Event Router](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html)\n  — the outbox pattern and how Debezium routes outbox rows to topics.\n\n\u003chttps://engineer.company/glossary/change-data-capture/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"How change data capture reads a database's own record of changes and passes it on, why it beats polling, and four ways it quietly goes wrong.","formats":{"gemtext":"https://engineer.company/glossary/change-data-capture/index.gmi","gopher":"https://engineer.company/glossary/change-data-capture/gophermap","html":"https://engineer.company/glossary/change-data-capture/","json":"https://engineer.company/glossary/change-data-capture/index.json","markdown":"https://engineer.company/glossary/change-data-capture/index.md","pdf":"https://engineer.company/glossary/change-data-capture/index.pdf"},"kind":"page","language":"en","title":"What is change data capture?","url":"https://engineer.company/glossary/change-data-capture/","version":1}