cdc

A mysql binlog parser, which has running stable for more then 36 months in our production environment on more than 10 mysql instances. It transforms binlog event(insert/delete/update) into json format data into rabbitmq or other mq(such as kafka), Based on this project: mysql-binlog-connector-java.

What is cdc?

change data capture, Key Features:

High availability, cluster deployment
Table level filter
Supports almost all mysql field parsing
Automatically save binlog file/position, smooth upgrade and restart
Configure centralized management
Operational status monitoring
Dynamically loading table configuration files
support split table config
Mysql binlog data order Guaranteed
Support a easy way to tap it into other mq (sucn as kafka,nsq,redis...)

dependency

Java 8 or above is required.
etcd 3.0 or above is required.

mysql 5.6+

open mysql binlog log

[mysqld]  
log-bin=mysql-bin
binlog-format=ROW
server_id=666

authorize mysql slave read permission

CREATE USER root IDENTIFIED BY '123'; 
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'root'@'%';
FLUSH PRIVILEGES;

Download

Maven

<dependency>
  <groupId>com.rong360</groupId>
  <artifactId>cdc</artifactId>
  <version>1.4.0</version>
</dependency>

Quick start

install etcd, start etcd

$ ./bin/etcd

Set the database configuration in etcd

etcdctl put cdc/master/admin/config/app/mysql/host 10.0.0.2
etcdctl put cdc/master/admin/config/app/mysql/port 3306
etcdctl put cdc/master/admin/config/app/mysql/username admin
etcdctl put cdc/master/admin/config/app/mysql/password 123

Start program

CdcClient client = new CdcClient("http://127.0.0.1:2379", "", "");
client.setInstance("admin");//"admin" is one of the ETCD configuration prefix elements
client.setWatchAllTable(true);
client.start();

Documentation

English

See DOUCUMENT for details.

简体中文

请见详细文档。

Connect with us

zhangtao@rong360.com

liuchi@rong360.com

rong360/cdc

cdc