CDC with Debezium: capturing changes from a customer's database without rescanning whole tables

A customer wants your agent to answer using the latest order status. The quick fix that comes to everyone’s mind is a cron job that runs SELECT * FROM orders every night and compares the result with the previous copy.

The first week goes fine. Then the table grows, the customer’s DBA asks why the database runs hot at 2am every night, and you discover you have never seen a single deleted row.

This is a familiar situation for FDEs: you need live data from a system you do not own, you cannot change the customer’s application code, and you must not slow down the production database. Change data capture (CDC) solves exactly this problem, and Debezium is an open-source tool built for the job.

Understanding CDC properly lets you talk to the customer’s DBA as a colleague rather than becoming a risk they have to watch. The sections below explain how Debezium works, with a concrete configuration example and the operational traps to close off before they become incidents.

Don’t query the table, read the log

Debezium describes itself as an open-source distributed platform for change data capture. It lets applications react to every insert, update and delete without touching the application that writes the data. For customers, that is the most valuable part: nobody has to change a running system.

The trick is that Debezium does not query tables. The PostgreSQL connector reads committed changes from the transaction log through logical decoding, using Postgres’s own streaming replication protocol. For MySQL, the connector reads the binlog and emits row-level events. A deleted row becomes an event too, something a table-diffing cron job never sees.

One detail makes the conversation with the DBA much easier. The pgoutput plug-in ships with PostgreSQL 10 and later, so you do not have to ask for any extra library to be installed on the customer’s server. In many enterprises every additional package on production means another approval cycle, and avoiding one can save you a week.

If it reads the log, why does it still need a snapshot?

This question is almost certain to come up. The answer is that the log is not the full history. The Debezium documentation states plainly that most Postgres servers do not keep the complete history of the database in the WAL, because the WAL is cleaned up after a while. MySQL is also commonly configured to purge binlogs after a set period.

Netflix’s DBLog paper describes the same problem: transaction logs usually do not contain the full history of changes, so capturing the log alone cannot synchronise an entire database. That is why, on first connection, the Postgres connector takes a consistent snapshot of the schemas and only then switches to reading the log from that point onwards.

The old snapshot model has two weaknesses with large tables. Streaming is blocked until the snapshot finishes, and if the snapshot is interrupted midway it cannot resume. Debezium’s incremental snapshot fixes this: the table is read in chunks while streaming keeps running, and you trigger it by writing a signal to a dedicated signal table.

To reconcile the data read in chunks with changes happening at the same time, Debezium borrows DBLog’s watermark idea: it inserts low and high watermarks into the transaction log around each chunk read.

According to the DBLog paper, this approach takes no table locks, so the impact on the source database is very small. That is the answer to give the DBA when they ask whether the snapshot will freeze the system.

Example: the orders table at a delivery company

Picture a customer that is a delivery company running Postgres 15, with a public.orders table your agent needs to track. You ask the DBA for a user with replication privileges and read access to the tables to be tracked, then create the signal table.

The signal table is how you give Debezium commands: to re-snapshot a table, all it takes is one INSERT into it. So agree with the DBA from the start who is allowed to write signals into this table, rather than asking for permission on the day you need a backfill. The connector configuration takes only a few lines:

{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db.customer.internal",
    "database.port": "5432",
    "database.user": "cdc_user",
    "database.dbname": "logistics",
    "topic.prefix": "customer_a",
    "plugin.name": "pgoutput",
    "table.include.list": "public.orders,public.debezium_signal",
    "signal.data.collection": "public.debezium_signal"
  }
}

When the connector starts, it snapshots orders and then switches to reading the WAL. An order moving from “out for delivery” to “delivered” becomes an update event. An order that is cancelled and deleted becomes a delete event. Your system receives only the rows that changed, not the whole table.

Three months later, the customer adds a column and you need to re-snapshot orders to backfill old data. Instead of stopping the pipeline, you write a signal:

INSERT INTO public.debezium_signal (id, type, data)
VALUES ('resnap-orders-1', 'execute-snapshot',
        '{"data-collections": ["public.orders"]}');

Debezium reads the table in chunks, streaming runs in parallel, and the watermarks handle reconciliation. The customer’s agent does not miss a minute of new data.

The trap is the replication slot

Gunnar Morling, author of a post dedicated to this risk, describes it clearly. When the connector stops, the slot is still there but inactive, and Postgres retains every WAL segment from the position where the slot stopped. The disk fills steadily until the customer’s database runs out of space.

You may have shut down the connector because a trial ended, but the slot still sits on the customer’s server. Morling recommends alerting when a slot retains more than 100 MB of WAL, while also monitoring free disk space. The check query is short:

SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained
FROM pg_replication_slots;

MySQL carries the opposite risk. If the connector stops for too long and the binlog at its position has been purged, the connector cannot resume and has to snapshot again.

The MySQL connector also records every DDL statement with its binlog position so it can reconstruct table structure at any point in time, so that schema history must be stored durably, not kept somewhere temporary.

In what order should you build it?

Start with customer discovery with the DBA: which version of Postgres or MySQL, how long the WAL or binlog is retained, who can create a replication user, and who is on call when the disk fills. The answer to the last question decides where your alerts go.

Next, agree on a specific list of tables rather than taking the whole database, create the signal table and settle who may write signals to it, then run a trial on staging with data volumes close to production.

Only once the slot alert is running, and has been tested by deliberately stopping the connector, should you switch on production. Finally, write a handover runbook that includes how to drop the slot when the project ends.

What are the most common mistakes?

The most common mistake is treating the slot as your own resource and forgetting it when tearing down a test environment. Almost as common is assuming the log holds enough history, skipping the snapshot, and then wondering why older orders are missing. Plenty of people also use the old-style snapshot on large tables and leave streaming blocked until it finishes.

One mistake is about communication: not telling the DBA up front that a stopped connector means WAL piles up. When they discover it for themselves at 3am, trust in the whole project is hard to win back.

How does this skill show up on a CV?

When reading job descriptions for FDE or data engineer roles, look for keywords such as CDC, Debezium, Kafka Connect and logical replication. On your CV, do not just write “used Debezium”. Write that you designed a CDC pipeline with incremental snapshots and per-slot WAL alerts, because that signals you have run it for real.

A good CDC pipeline is one the customer almost forgets is running. Your job is to make sure the only thing they remember is that the data is always fresh, not the disk that filled up.