benbjohnson/litestream

Add -once and -force-snapshot arguments to replicate command

Opened this issue · 1 comments

hifi commented

Related to #436 I spoke with someone either in some issue or on Slack sometime ago that there's a need for Litestream to be able to be executed periodically instead of continuously.

Our use case is to ensure the local database has been fully replicated to remote storage before we remove it to prevent data loss when we are intentionally moving the process out from the host it was running on and starting it on another. The current workaround is to use exec but it's not always reliable enough and for scheduled maintenance it's sometimes a lot faster to upload a snapshot and download it with restore instead of using the WAL chunks.

My proposal is that the replicate command gains -once to make it only replicate anything it has not replicated before on startup if the WAL hasn't been rolled out by the application process and -force-snapshot to force a snapshot to be taken on startup regardless if -once was used or not. This allows us to use them in combination to do a final backup of a database and ensure it's consistent by running restore as well.

The -once argument needs force all the maintenance tasks as well or they need to be behind another flag like --force-retention

This change would also allow cron style executing of Litestream given two preconditions are met:

  • application uses PRAGMA wal_autocheckpoint = 0
  • application uses SQLITE_FCNTL_PERSIST_WAL
  • Litestream can issue a passive checkpoint during at least some of the runs

Of course in cron style execution the write pressure must be extremely low not to block Litestream checkpoints and to not grow the WAL ridiculously but for some workloads this might make sense - or they just want to take full snapshots once in a while and have a binary that can do encrypted uploads and downloads to a bucket and always get the latest one on restore with

I will implement something like this soon in downstream but it'd be great if I could architect it to be upstream compatible given it would be accepted in some form.

hifi commented

Created a draft PR. It sort of works and I'm interested in comments before going further with it.