Browse documentation▾

Documentation

Everything you need to replicate MySQL into BigQuery: setup for each host, how change data capture works, and how to keep BigQuery costs low.

Start here

Getting started

Replicate MySQL tables into BigQuery with Quayen in about 15 minutes: connect MySQL, connect BigQuery, choose tables.

Concepts

How MySQL to BigQuery CDC works

How change data capture works: consistent snapshot, binary log streaming, change history and refreshes, and how ordering and deletes are handled.

Setup

MySQL setup for change data capture

MySQL settings required for CDC to BigQuery: row-based binary logging, full row images, binlog retention, a least-privilege user and SSH bastions.

Setup

BigQuery setup

Set up BigQuery for MySQL replication: project and billing, a service account with BigQuery Data Editor and Job User roles, a JSON key, and dataset location.

Concepts

Table layout in BigQuery

The BigQuery tables created for each MySQL table: the up-to-date copy and the change history, their system columns, partitioning and example queries.

Guides

Keeping BigQuery costs low

How BigQuery bills CDC MERGE jobs and how to minimise cost: insert-only merges, clustering, partitioning, refresh interval and change-history retention.

Learn

What is change data capture (CDC)?

Change data capture explained: how CDC works, log-based vs query-based CDC, common CDC tools, and how MySQL change data capture uses the binary log.

Learn

MySQL binlog_format and binlog_row_image explained

What MySQL's binlog_format (ROW, STATEMENT, MIXED) and binlog_row_image (FULL, MINIMAL, NOBLOB) settings do, which values CDC needs, and how to check them.

Learn

How to sync MySQL deletes to BigQuery

Why scheduled exports and timestamp-based syncs miss deleted rows, and how log-based CDC applies MySQL deletes to BigQuery while keeping a history of them.

Learn

How to connect MySQL to BigQuery: 5 ways, including real time

Five ways to connect, sync or migrate MySQL to BigQuery, compared on freshness, deletes and load, with real-time replication set up step by step.

Learn

The best tools to replicate MySQL to BigQuery, compared

Google Cloud Datastream, Fivetran, Airbyte, Hevo, Estuary, Debezium and Quayen compared on latency, pricing model, deletes and change history, with when each one fits.

Learn

Replicate a private RDS MySQL database to BigQuery

How to stream an Amazon RDS or Aurora MySQL database that has no public endpoint into BigQuery through an SSH bastion, with the exact security group rules.

Learn

How long to keep MySQL binary logs for CDC

Why binary log retention decides whether MySQL CDC can recover from an outage, what to set it to, and how to change it on RDS, Aurora, Cloud SQL and Azure.

Learn

Common MySQL CDC errors and how to fix them

Fixes for common MySQL CDC errors: purged binary logs (1236), missing replication privileges (1227), wrong binlog format or row image, and missing primary keys.

Learn

Partitioning BigQuery tables replicated from MySQL

When to partition the BigQuery copies of MySQL tables, which column and granularity to choose, and how to change partitioning on a table that already exists.

Learn

Nightly MySQL exports vs change data capture: when to switch

How nightly dumps and scheduled exports compare with real-time CDC for getting MySQL into BigQuery, and the signs it's time to switch.

Learn

BigQuery vs MySQL: differences and when to use each

BigQuery vs MySQL compared on storage, speed, transactions, cost and scale: what each is good at, and why most teams run both.

Learn

Google Cloud Datastream pricing explained, with examples

How Datastream bills MySQL CDC and backfill per GiB processed, its 2,500, 5,000 and 10,000 GiB tiers, the free backfill allowance, and worked monthly examples.

Learn

Google Cloud Datastream alternatives for MySQL to BigQuery

When Google Cloud Datastream isn't the right fit for MySQL to BigQuery, and the alternatives: Quayen, Fivetran, Airbyte, Estuary, Hevo, Debezium and federated queries.

Learn

How to enable binary logging on Cloud SQL for MySQL

Turn on the binary log in Cloud SQL for MySQL by enabling point-in-time recovery (console or gcloud), check it with SHOW VARIABLES, and set retention for CDC.

Learn

Replicate a private-IP Cloud SQL MySQL instance to BigQuery

Stream a Cloud SQL for MySQL instance that only has a private IP into BigQuery through a small SSH bastion VM in the same VPC, without adding a public IP.

Learn

Debezium to BigQuery: streaming MySQL changes and what you run

The ways to get Debezium's MySQL change events into BigQuery (Kafka Connect sink, Debezium Server with Pub/Sub), how updates and deletes are applied, and what you operate.