Browse documentation▾
Documentation
Everything you need to replicate MySQL into BigQuery: setup for each host, how change data capture works, and how to keep BigQuery costs low.
Start here
Getting started
Replicate MySQL tables into BigQuery with Quayen in about 15 minutes: connect MySQL, connect BigQuery, choose tables.
Concepts
How MySQL to BigQuery CDC works
How change data capture works: consistent snapshot, binary log streaming, change history and refreshes, and how ordering and deletes are handled.
Setup
MySQL setup for change data capture
MySQL settings required for CDC to BigQuery: row-based binary logging, full row images, binlog retention, a least-privilege user and SSH bastions.
Setup
BigQuery setup
Set up BigQuery for MySQL replication: project and billing, a service account with BigQuery Data Editor and Job User roles, a JSON key, and dataset location.
Concepts
Table layout in BigQuery
The BigQuery tables created for each MySQL table: the up-to-date copy and the change history, their system columns, partitioning and example queries.
Guides
Keeping BigQuery costs low
How BigQuery bills CDC MERGE jobs and how to minimise cost: insert-only merges, clustering, partitioning, refresh interval and change-history retention.
Learn
What is change data capture (CDC)?
Change data capture explained: log-based vs query-based CDC, why reading the MySQL binary log is the most reliable way to replicate, and deletes.
Learn
MySQL binlog_format and binlog_row_image explained
What MySQL's binlog_format (ROW, STATEMENT, MIXED) and binlog_row_image (FULL, MINIMAL, NOBLOB) settings do, which values CDC needs, and how to check them.
Learn
How to sync MySQL deletes to BigQuery
Why scheduled exports and timestamp-based syncs miss deleted rows, and how log-based CDC applies MySQL deletes to BigQuery while keeping a history of them.
Learn
How to replicate MySQL to BigQuery in real time
The five ways to get MySQL data into BigQuery, compared on freshness, deletes, load and effort, and how to set up real-time replication step by step.
Learn
Replicate a private RDS MySQL database to BigQuery
How to stream an Amazon RDS or Aurora MySQL database that has no public endpoint into BigQuery through an SSH bastion, with the exact security group rules.
Learn
How long to keep MySQL binary logs for CDC
Why binary log retention decides whether MySQL CDC can recover from an outage, what to set it to, and how to change it on RDS, Aurora, Cloud SQL and Azure.
Learn
Common MySQL CDC errors and how to fix them
Fixes for common MySQL CDC errors: purged binary logs (1236), missing replication privileges (1227), wrong binlog format or row image, and missing primary keys.
Learn
Partitioning BigQuery tables replicated from MySQL
When to partition the BigQuery copies of MySQL tables, which column and granularity to choose, and how to change partitioning on a table that already exists.
Learn
Nightly MySQL exports vs change data capture: when to switch
How nightly dumps and scheduled exports compare with real-time CDC for getting MySQL into BigQuery, and the signs it's time to switch.