topling/toplingdb

RocksIterator can not get the data on secondary instance

linking12 opened this issue · 0 comments

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

One of Rocksdb secondary instance can not get the correct data after 16 hours.

Expected behavior
startup one primary and 3 secondary instances on k8s. every instances can get the same result for the same query.

Actual behavior
after 16 hours, primary and 2 secondary instances got 1000 records for user. one instance got 500 records.

Steps to reproduce the behavior
startup one primary and 3 secondary instances on k8s.
secondary instances invoke tryCatchUpWithPrimary every 3 seconds.
everything wroks fine at this time, can get the items correctly on the 3 secondary instances
after one day, there is a secondary instance become abnormal, can not get the whole data comparing to other instances, no exception throwed by tryCatchUpWithPrimary. example: other 3 instance(primary and other 2 secondary) can get 1000 records, but this instance can only get 500 records. and can not get the newest item inserted by primary as well
first saw on Java RocksdbJni 8.8.1, then downgrade to 6.29.4, still the same.
env:
Java + Spring Boot
aws k8s/aws efs
Rocksdb version: 6.29.4
Java RocksdbJni: 6.29.4.1

have any suggestion for this issue?

这个issue在RocksDB提了下,在国人社区也提下,请问下有碰到类似这种情况么?