Improve Snapshoting
Closed this issue · 0 comments
kingster commented
Currently, snapshotting takes a full backup and blocks writes for a very long time
dkv[28902]: 2021/06/28 15:27:03 nexus.raft: [Node b181b0534fcfb32f] start snapshot [applied index: 50001 | last snapshot index: 0]
dkv[28902]: 2021/06/28 15:31:13 nexus.raft: [Node b181b0534fcfb32f] compacted log at index 45001
that's a whopping 5 minutes for just a 11G data
$ du -sh /var/lib/dkv/data
11G /var/lib/dkv/data
Snapshots need to be faster/non-blocking.
- Streaming Snapshots
Also snapshotting should not load the entire data in-memory to avoid oom panics #64