mydit: A Scala repository from ngocdaothanh

Mydit: MySQL to MongoDB data replicator

You can use it to implement transaction feature for programs using MongoDB: Write transaction data to MySQL, then read from MongoDB.

The download looks like this:

config/
  application.conf
  logback.xml
lib/
  <.jar files>
script/
  start
  start.sh

Java 6+ is required. Run:

script/start

How Mydit works

MySQL ---> Mydit ---> MongoDB

mysql-binlog-connector-java is used to get a stream of row events from MySQL.

The events are then replicated to MongoDB, using WriteConcern ACKNOWLEDGED.

Info about MySQL's binlog file name and replication position is saved in MongoDB. When the replicator is restarted, it knows where to continue the replication.

Error handling:

MySQL: When there's problem connecting to MySQL, the replicator will try to reconnect.
MongoDB: When there's problem replicating to MongoDB, the replicator will queue the data and try to replicate it later. If the queue size exceeds the configured threshold, the replicator process will exit.

MySQL prerequisites

See How to Set Up Replication.

MySQL's binlog_format must be set to ROW. Note that RDS MySQL's binlog_format is set to MIXED by default. If you use RDS you need to create a parameter group and set it to the MySQL instance. You may need to restart the instance for the setting in the parameter group to take effect.

The user to connect to the MySQL server must have privileges REPLICATION SLAVE and REPLICATION CLIENT.

Don't forget to set server-id, otherwise slaves cannot connect.

Typically, my.cnf should look like this:

[mysqld]
server-id        = 1
binlog_format    = row
log_bin          = mysql-bin.log
expire_logs_days = 10
max_binlog_size  = 100M

Things Mydit doesn't do

If there are indexes in your MySQL databases, you have to manually create MongoDB indexes yourself.
When you change MySQL database schema, you need to do apply the change at MongoDB side yourself.

Config

See config/application.conf.

If you want to set, for example, MySQL password from system environment variable, modify config/application.conf like this:

...
password = "${MYSQL_PASSWORD}"
...

To config JVM memory, see script/start and script/start.bat.

Log

When the program runs, log will be output to log/mydit.log. The log is rotated daily by default. See config/logback.xml.

Start with Supervisor

Supervisor is a very easy to use tool to start program at OS boot and restart program automatically when it stops abnormally.

See Ubuntu tutorial.

/etc/supervisor/conf.d/mydit.conf example:

[program:mydit]
command=/home/ubuntu/opt/mydit-1.0/script/start
directory=/home/ubuntu/opt/mydit-1.0
autostart=true
autorestart=true
startretries=3
stderr_logfile=/home/ubuntu/opt/mydit-1.1/supervisord.err.log
stdout_logfile=/home/ubuntu/opt/mydit-1.1/supervisord.out.log
user=ubuntu

Command examples:

sudo supervisorctl reread
sudo supervisorctl update

sudo supervisorctl status mydit
sudo supervisorctl stop mydit
sudo supervisorctl start mydit

Bootstrap or rebootstrap MongoDB from existing MySQL data

There are two scenarios:

MongoDB is empty and you want to populate it with existing MySQL data.
Replication binlog position is too old and MySQL has deleted the old binlog file.

To make MongoDB data to be updated to the latest MySQL data, you can do like this:

Alter MySQL tables to include a dummy additional field.
Update MySQL tables to set value to that field. This will force all rows in the tables to be replicated to MongoDB.
Delete the field in both MySQL and MongoDB.

ngocdaothanh/mydit