What is change data capture (CDC)?

Change data capture (CDC) is a way of copying data by capturing each change (insert, update or delete) as it happens, instead of repeatedly re-copying whole tables. It keeps a destination such as BigQuery in sync with a source database with low delay and little load on the source.

Three ways to capture changes

• Full reloads. Copy the whole table on a schedule. Simple, but slow and expensive for large tables, and the copy is only as fresh as the last run.
• Query-based (polling). Repeatedly query for rows with a newer updated_at. It misses deletes entirely, misses rows whose timestamp isn't updated, and puts query load on the database.
• Log-based. Read the database's own transaction log, in MySQL the binary log. Every committed change is recorded in order, including deletes, and reading it barely affects the database. This is the method Quayen uses.

Why deletes matter

Polling can't see a row that no longer exists, so polled copies slowly fill with rows that were deleted in MySQL. Log-based CDC sees the delete event and removes the row from the BigQuery copy, while the change history still records that it existed.

Snapshot plus stream

A CDC pipeline starts with a consistent snapshot of existing rows, records the exact log position, then streams changes from that position, so nothing is lost or applied twice. See how it works.