MongoDB to MongoDB live sync: "resume timeout reached"

[madankumark]

Bug report

I'm seeing multiple issues when trying to live sync a large MongoDB database(roughly around 25GB and having nearly 115 collections)
Initially sync went fine however hit many issues going forward:

1. Timeout reached waiting for Out channels to clear (context deadline exceeded)
   I added write_timeout config variable with 600 seconds. However after this i hit the below issue.
2. Resume timeout reached
   After the above error whenever i try running transporter it ends up in resume timeout error and stops there.

Other issues i see are:

1. "flush error, Write batch sizes must be between 1 and 1000. Got 1101 operations."
2. msg="unable to sample document, not found" collection=

Relevant pipeline.js:

var source = mongodb({
"uri": "${MONGODB_SOURCE_URI}"
"timeout": "60s",
"tail": true,
// "ssl": false,
// "cacerts": ["/path/to/cert.pem"],
// "wc": 1,
// "fsync": false,
// "bulk": false,
// "collection_filters": "{}",
// "read_preference": "Primary"
})

var sink = mongodb({
"uri": "${MONGODB_SINK_URI}"
"timeout": "60s",
// "tail": false,
"ssl": true,
"cacerts": ["${CERT_PATH}"],
// "wc": 1,
// "fsync": false,
"bulk": true,
// "collection_filters": "{}",
// "read_preference": "Primary"
})

t
.Config({"write_timeout": "600s", "log_dir":"${COMMIT_LOG_PATH}"})
.Source("source", source, "/./")
.Save("sink", sink, "/./")

System info:

• Transporter version
  0.5.2
• OS
  Amazon Linux AMI release 2017.09
• DB version(s)
  source mongo: 3.0.5
  sink mongo: 3.4.9

Reproducible Steps:

1. Run transporter to live sync huge database with above pipeline config.

What did you expect to happened?

1. Live sync should have resumed.

What actually happened?

1. resume is timed out