Integrations

Self-hosted MySQL to BigQuery, in real time

Replicate a self-hosted MySQL 8 server (on a VM, bare metal or Kubernetes) into BigQuery with change data capture. Add four lines to my.cnf, restart, and stream every change within seconds.

Setup time
About 15 minutes
Restart needed
Yes: MySQL must be restarted once after enabling the binary log.
Binary log retention
binlog_expire_logs_seconds = 259200

Set up Self-hosted MySQL 8 for change data capture

The in-app guide shows each of these with exact clicks and copy-paste commands, then tests the connection. Full reference: MySQL setup for change data capture.

  1. 1

    Edit the MySQL configuration

    Open my.cnf (often /etc/mysql/mysql.conf.d/mysqld.cnf or /etc/my.cnf) and, under [mysqld], set log_bin = mysql-bin, binlog_format = ROW, binlog_row_image = FULL and binlog_expire_logs_seconds = 259200.

  2. 2

    Restart MySQL

    Restart the service, for example sudo systemctl restart mysql.

  3. 3

    Accept outside connections

    Set bind-address = 0.0.0.0 (it is often 127.0.0.1) and open port 3306 to our IP addresses only in your firewall.

  4. 4

    Create a read-only replication user

    GRANT SELECT on your database plus REPLICATION SLAVE and REPLICATION CLIENT.

  5. 5

    Connect BigQuery and pick tables

    Upload a service-account key, choose the tables, and start. Existing rows are copied first; after that every insert, update and delete arrives within seconds. See BigQuery setup.

What you get in BigQuery

  • An up-to-date copy of each table with the same primary key, optionally partitioned by date.
  • A change history of every insert, update and delete in a separate dataset, kept as long as you choose.
  • New columns added automatically, and per-table freshness you can monitor.

Related guides

Self-hosted MySQL 8 questions

Which MySQL versions are supported?

MySQL 8 with row-based binary logging.

Can I replicate from a replica instead of the primary?

Yes, if the replica writes its own binary log (log_replica_updates = ON) in ROW format.

How do I connect Self-hosted MySQL 8 to BigQuery?

Follow the steps above to turn on the binary log and allow access, create a read-only replication user, then upload a BigQuery service-account key and choose the tables. Existing rows are copied once and every later insert, update and delete is replicated. Most teams finish in about 15 minutes.

How quickly do Self-hosted MySQL 8 changes reach BigQuery?

Each change reaches a change-history table in BigQuery within seconds. The up-to-date copy of each table refreshes on a schedule you choose, from every minute to every hour (every 5 minutes by default).

Can I migrate Self-hosted MySQL 8 to BigQuery once instead of syncing it?

Yes: export the tables with mysqldump or as CSV and load them with bq load, after converting the schema with the free MySQL to BigQuery schema converter. A one-time copy goes stale as soon as MySQL changes, so use replication if reports need current data.

Your MySQL data in BigQuery today

Free during early access. Set it up yourself in about 15 minutes, or we'll do it with you on a call.