Log compaction
Opened this issue · 2 comments
Hi,
Our implementation is pretty complete now - few more hurdles!
We are running raft in embedded terminals that communicate locally on a wired/wireless network. typically there would be 2-10 terminals participating in a cluster.
Each terminal applies entries from the log via applylog() to their local sqlite db but there is no mechanism at present to ever truncate that log. As the system is used it will become unmanageably large - the log is loaded on startup so we can provide log entries to followers if necessary, but loading this log is already becoming slow.
In this application there is a concept of agreeing on a time period. This is a point at which all earlier log entries are redundant; they might be applied to the local db but they are no longer relevant anyway. For example, entries from yesterday might be unimportant and candidates for removing.
My plan to compact the log is to append a log entry when this period is reached, this log entry will include the entry_idx. The other terminals commit this, when they apply they truncate all entries up to and including this entry.
Thoughts?
Neil
Hello,
In your case, you're approach to log compaction sounds good. I have to iron out the documentation to help make it more clear how you'll get the library to poll (ie. truncate) the redundant entries. This shouldn't take too long.
I'm currently working on an example of log compaction. I'll be adding this to ticketd. This'll be done in 2-4wks.
You might want to get hints from https://github.com/ar104/raft which is a fork that implements log compaction directly into the library.
Thanks