Self-hosted MySQL to BigQuery, in real time
Replicate a self-hosted MySQL 8 server (on a VM, bare metal or Kubernetes) into BigQuery with change data capture. Add four lines to my.cnf, restart, and stream every change within seconds.
- Setup time
- About 15 minutes
- Restart needed
- Yes: MySQL must be restarted once after enabling the binary log.
- Binary log retention
- binlog_expire_logs_seconds = 259200
Set up Self-hosted MySQL 8 for change data capture
The in-app guide shows each of these with exact clicks and copy-paste commands, then tests the connection.
- 1
Edit the MySQL configuration
Open my.cnf (often /etc/mysql/mysql.conf.d/mysqld.cnf or /etc/my.cnf) and, under [mysqld], set log_bin = mysql-bin, binlog_format = ROW, binlog_row_image = FULL and binlog_expire_logs_seconds = 259200.
- 2
Restart MySQL
Restart the service, for example sudo systemctl restart mysql.
- 3
Accept outside connections
Set bind-address = 0.0.0.0 (it is often 127.0.0.1) and open port 3306 to our IP addresses only in your firewall.
- 4
Create a read-only replication user
GRANT SELECT on your database plus REPLICATION SLAVE and REPLICATION CLIENT.
- 5
Connect BigQuery and pick tables
Upload a service-account key, choose the tables, and start. Existing rows are copied first; after that every insert, update and delete arrives within seconds. See BigQuery setup.
What you get in BigQuery
- An up-to-date copy of each table with the same primary key, optionally partitioned by date.
- A change history of every insert, update and delete in a separate dataset, kept as long as you choose.
- New columns added automatically, and per-table freshness you can monitor.
Self-hosted MySQL 8 questions
Which MySQL versions are supported?
MySQL 8 with row-based binary logging.
Can I replicate from a replica instead of the primary?
Yes, if the replica writes its own binary log (log_replica_updates = ON) in ROW format.
Your MySQL data in BigQuery today
Free during early access. Set it up yourself in about 15 minutes, or we'll do it with you on a call.