torodb/stampede

ToroDB shuts down unexpectedly

fer2d2 opened this issue · 2 comments

fer2d2 commented

Hi @teoincontatto ,

When stampede is running for some time, it shuts down suddenly. Taking a look at the logs, you can see the following messages:

MongoDB Logs

Jan  3 21:22:21 10.255.0.7 docker[mongodb][25132]: 2019-01-03T21:22:21.596+0000 I -        [conn332518] Assertion: 16089:Cannot kill pinned cursor: 13122723805
Jan  3 21:22:21 10.255.0.7 docker[mongodb][25132]: 2019-01-03T21:22:21.596+0000 I COMMAND  [conn332518] killcursors local.oplog.rs keyUpdates:0 writeConflicts:0 exception: Cannot kill pinned cursor: 13122723805 code:16089 numYields:0 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, oplog: { acquireCount: { r: 1 } } } 0ms

Stampede logs

torodb-stampede_1_39129f78021c | 2019-01-03T09:20:50.846 INFO  REPL       - syncing from: mongodb:27017
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:50.846 INFO  REPL       - Topology service started
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:50.903 INFO  REPL       - Database is consistent.
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:50.904 INFO  REPL       - Replication service started
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:50.906 INFO  LIFECYCLE  - ToroDB Stampede is now running
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:50.907 INFO  REPL       - Starting SECONDARY mode
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:51.139 INFO  REPL       - Reading from mongodb:27017
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:53.576 WARN  REPL       - Oplog replication stream finished exceptionally: null
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:53.592 ERROR REPL       - Catched an error on the replication layer. Escalating it
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:53.592 ERROR LIFECYCLE  - Error reported by replication supervisor. Stopping ToroDB Stampede
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:53.599 INFO  LIFECYCLE  - Shutting down ToroDB Stampede
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:53.612 INFO  REPL       - Shutting down replication service
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:55.003 INFO  REPL       - Topology service shutted down
torodb-stampede_1_39129f78021c | 2019-01-03T09:20:55.017 INFO  REPL       - Replication service shutted down

More info

MongoDB

root@b889778dc962:/# mongod --version
db version v3.2.21
git version: 1ab1010737145ba3761318508ff65ba74dfe8155
OpenSSL version: OpenSSL 1.0.1t  3 May 2016
allocator: tcmalloc
modules: none
build environment:
    distmod: debian81
    distarch: x86_64
    target_arch: x86_64
  • Using user and password auth.

ToroDB

  • Version 1.0.0 (latest).

What can I do to get more information about this error and fix it?

Thanks in advance.

fer2d2 commented

Could this error be related with this one?

teoincontatto commented

Hi @fer2d2,

Can you enable stacktrace with this log4j2.xml file:

torodb-stampede --log4j2-file log4j2.xml