qoollo/bob

Make record deletion recoverable with BobAliensRecovery

Closed this issue · 0 comments

Currently, if node is not available during deletion, the data on it would not be marked as deleted even after its recovery. In this case, the data in the cluster becomes inconsistent: the record on one replica is deleted, but not on the other. We need to support recovery.

We can do it by making deletion explicitly visible for Bob and always write a special deletion record to the active blob in aliens. Pearl should return the deleted status when trying to read a deleted record, and Bob should stop scanning older timestamps if it receives that status. So BobAliensRecovery will recover blobs from aliens that contains that deleted record and Bob stop scanning if it sees it.

To achieve this on the Pearl side we should introduce deleted flag on the index level. This will allow deletion to be reported without reading the data from the blob, thus quickly returning deleted status. Additionally, deleted status on index level will simplify blob compaction feature in future