/mongosync

mongosync is simple && useful tool to sync data between mongo replicaSet, it is developed by Qihoo's DBA and infrastructure team

Primary LanguageC++

README

中文wiki前往

1.Introduction

This simpe program is for transfering data between two mongo nodes, including:

  • data cloning;
  • oplog syncing(applying one mongo databbase's oplog to another mongo database) ;
  • oplog cloning(not applying, just storing the oplogs from source mongo database to another one)

2.Usage

  • Get the source code

    $ git clone https://github.com/Qihoo360/mongosync.git
  • Compile

    Enter the source code and typewrite:

    $ make

    after that, a "mongosync" named execution file appears in the $(MONGOSYNC_ROOT)/output directory, and also a "mongosync.conf" named config file

    SP:

    a. maybe some depenncies not installed in your machine, install them according to the tips(mostly sons, boost and ssl is needed)

    b. don't use c++11 to compile this program, otherwise will occur error

  • Exectution

    The program can launches with command line or config file,and the specified usage can be obtained by:

    $ cd output
    $ ./mongosync --help
    Follow is the mongosync-surpported options:
    --help                   to get the help message
    -c conf.file             use config file to start mongosync
    --src_srv arg            the source mongodb server's ip port
    --src_user arg           the source mongodb server's logging user
    --src_passwd arg         the source mongodb server's logging password
    --src_auth_db arg        the source mongodb server's auth db
    --src_use_mcr            force source connection to use MONGODB-CR password machenism
    --dst_srv arg            the destination mongodb server's ip port
    --dst_user arg           the destination mongodb server's logging user
    --dst_passwd arg         the destination mongodb server's logging password
    --dst_auth_db arg        the destination mongodb server's auth db
    --dst_use_mcr            force destination connection to use MONGODB-CR password machenism
    --db arg                 the source database to be cloned
    --dst_db arg             the destination database
    --coll arg               the source collection to be cloned
    --dst_coll arg           the destination collection
    --oplog                  whether to sync oplog
    --raw_oplog              whether to only clone oplog
    --op_start arg           the start timestamp to sync oplog
    --op_end arg             the start timestamp to sync oplog
    --dst_op_ns arg          the destination namespace for raw oplog mode
    --no_index               whether to clone the db or collection corresponding index
    --filter arg             the bson format string used to filter the records to be transfered

3.Performance

Rough testing under the situation:

  • Ping from this program's exectution's machine to the mongo db server's machine(two machines) is about 1.5ms
  • total about 36,400,000 documents;
  • database size about 1.09GB

transfering time is about 379 seconds(even qps: 96.04k).

SP: The bottleneck is writing, So using multiple threads(multi-conn) to accelerate the speed of that(but also limited by database server's writing speed).