Learn
Debezium to BigQuery: streaming MySQL changes and what you run
The ways to get Debezium's MySQL change events into BigQuery (Kafka Connect sink, Debezium Server with Pub/Sub), how updates and deletes are applied, and what you operate.
Debezium reads the MySQL binary log and turns every insert, update and delete into a change event. It doesn't write BigQuery tables itself: you choose how events travel and how they become tables.
Option 1: Kafka and a BigQuery sink connector
The classic setup: the Debezium MySQL connector runs in Kafka Connect and writes one Kafka topic per table, and a BigQuery sink connector (for example the open-source kafka-connect-bigquery) reads the topics. Sink connectors can append every event, or upsert and delete by primary key by staging rows and running periodic MERGE statements.
You run Kafka brokers, Kafka Connect workers, the schema registry or converters, and monitoring for all of them.
Option 2: Debezium Server with Pub/Sub
Debezium Server runs the connector without Kafka and can send events to Google Cloud Pub/Sub. A Pub/Sub BigQuery subscription then writes each message as a row. That gives you an append-only change log; for an up-to-date copy you add views or scheduled MERGE jobs that keep the latest version of each key and drop deleted ones.
What you have to get right
- Deletes: Debezium emits a delete event (and optionally a tombstone); the sink must remove the row, not just record it. See syncing deletes.
- Ordering: apply changes per key in binlog order, or an older update can overwrite a newer one.
- Schema changes: new columns must reach the BigQuery table before rows that use them.
- Initial snapshot: Debezium snapshots tables first; large tables need tuning and enough binlog retention to finish.
- BigQuery cost: frequent
MERGEs scan the target table each time; partition and cluster it (BigQuery costs).
When a managed service is simpler
If you only need MySQL in BigQuery, not an event stream for other consumers, a managed service removes the Kafka and merge layers. Quayen does the whole path, with an up-to-date table and a change history for each MySQL table: Quayen vs Debezium.