Context deadline exceeded in Sharding

[rajavignesh]

I have 2 shards (2 nodes, 1 arbiter), Config server with 3 node replica set and 1 mongos instance.
When running keyhole for 5 mins, data getting loaded but getting below error and not progressing further.

Command
./keyhole --duration 5 --conn 5 --tps 5 "mongodb://mongodba:mongodba@mvp.sadc-mongo-sbx01:20007"

Error Message
Begin a load test [y/N]: y
2021-05-12T01:05:48-04:00 I Duration in minute(s):5
2021-05-12T01:05:48-04:00 I Total TPS: 25 (5 tps/conn * 5 conns), duration: 5 (mins)
2021-05-12T01:05:48-04:00 I Sharding collection:_KEYHOLE_88800.__examples
2021-05-12T01:05:48-04:00 I splitting chunks...
2021-05-12T01:05:48-04:00 I moving chunks...
2021-05-12T01:05:48-04:00 I moving map[email:O] from rs1 to rs0
2021-05-12T01:06:19-04:00 I context deadline exceeded
2021-05-12T01:06:49-04:00 I server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }
2021-05-12T01:07:20-04:00 I Thread1existing withserver selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }
2021-05-12T01:07:20-04:00 I Thread0existing withserver selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }
2021-05-12T01:07:20-04:00 I Thread4existing withcontext deadline exceeded
2021-05-12T01:07:20-04:00 I Thread2existing withserver selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }
2021-05-12T01:07:20-04:00 I Thread3existing withserver selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }

[simagix]

It's a DNS and connectivity issue. Make sure your mongod is accessible by keyhole. Can't help with your environment issues.

[rajavignesh]

I am connecting to the mongos instance and the data is getting loaded from a keyhole. Will the keyhole try connecting to ReplicaSet when the connection string is for mongos?

[simagix]

yes, it connects to mongod to collect stats.