/mongosync

mongosync is simple && useful tool to sync data between mongo replicaSet, it is developed by Qihoo's DBA and infrastructure team

Primary LanguageC++

README

中文wiki前往

1.Introduction

This simpe program is for transfering data between two mongo nodes, including:

  • data cloning;
  • oplog syncing(applying one mongo databbase's oplog to another mongo database) ;
  • oplog cloning(not applying, just storing the oplogs from source mongo database to another one)

2.Usage

  • Get the source code

    $ git clone https://github.com/Qihoo360/mongosync.git

  • Compile

    Enter the source code and typewrite:

    $ make

    after that, a "mongosync" named execution file appears in the $(MONGOSYNC_ROOT)/output directory, and also a "mongosync.conf" named config file

    SP:

    a. maybe some depenncies not installed in your machine, install them according to the tips（mostly sons, boost and ssl is needed）

    b. don't use c++11 to compile this program, otherwise will occur error

  • Exectution

    The program can launches with command line or config file，and the specified usage can be obtained by:

    $ cd output
$ ./mongosync --help
Follow is the mongosync-surpported options:
--help                   to get the help message
-c conf.file             use config file to start mongosync
--src_srv arg            the source mongodb server's ip port
--src_user arg           the source mongodb server's logging user
--src_passwd arg         the source mongodb server's logging password
--src_auth_db arg        the source mongodb server's auth db
--src_use_mcr            force source connection to use MONGODB-CR password machenism
--dst_srv arg            the destination mongodb server's ip port
--dst_user arg           the destination mongodb server's logging user
--dst_passwd arg         the destination mongodb server's logging password
--dst_auth_db arg        the destination mongodb server's auth db
--dst_use_mcr            force destination connection to use MONGODB-CR password machenism
--db arg                 the source database to be cloned
--dst_db arg             the destination database
--coll arg               the source collection to be cloned
--dst_coll arg           the destination collection
--oplog                  whether to sync oplog
--raw_oplog              whether to only clone oplog
--op_start arg           the start timestamp to sync oplog
--op_end arg             the start timestamp to sync oplog
--dst_op_ns arg          the destination namespace for raw oplog mode
--no_index               whether to clone the db or collection corresponding index
--filter arg             the bson format string used to filter the records to be transfered

3.Performance

Rough testing under the situation:

  • Ping from this program's exectution's machine to the mongo db server's machine(two machines) is about 1.5ms；
  • total about 36,400,000 documents；
  • database size about 1.09GB

transfering time is about 379 seconds(even qps: 96.04k).

SP: The bottleneck is writing, So using multiple threads(multi-conn) to accelerate the speed of that(but also limited by database server's writing speed).