New Sentry logic fails if replay log hits EOFError

[mbucc]

1. start prevalyer
2. log a transaction (Sentry.serialId = 1)
3. restart prevalyer, hit EOF error (Log.serialId incremented b/f exception thrown)
4. log a transaction (Sentry.serialId = 3)
5. restart prevalyer (BOOM! assertion failure)

In Log.loadInitState(), don't increment serialId until after transaction loads successfully from pickle file.

[mbucc]

So, this is interesting.

How should system behave if there is an exception loading the Sentry transaction from the Log?

Crash immediately? (My fix does not deal with this case.)

[mbucc]

i think there are five cases:

1. both trx and sentry trx load
2. eoferror loading trx
3. other error loading trx
4. eoferror loading sentry trx
   5 other error loading sentry trx

cases 3, 4 and 5 are currently ignored.

[PetrGlad]

// I am sorry for not answering earlier.
If there's any error reading transaction log one should not continue - the data is corrupted. You may opt to resume working from last known good state or just stop with error immediately. One more option might be to use a snapshot that is later than the failed log entry but if it existed prevayler would not read the log anyway.
If you decide to ignore broken tail of log then there we need some logic to fix that up somehow - say truncate that log at last known position, erase or quarantine all subsequent logs, make snapshot immediately and start new log from the next transaction.

Maybe adding record boundaries is not the best idea but at least it helps to ensure that only whole transaction is read and verify transactions order so one does not rely on file names only.