Support for borg backups
Opened this issue · 0 comments
ml-evs commented
In https://github.com/datalab-industries/datalab-ansible-terraform I've been working on using borg and borgmatic for automated, encrypted, compressed and de-duplicated snapshot backups. This is slightly harder to setup than the simple filesystem dump, but greatly enhances resilience and massively reduces storage space. There is direct support for backing up MongoDB in borgmatic too.
I propose to add this as another backup type directly in datalab, so that datalab can:
- check backup health of remote borg backups
- restore from a borg snapshot
- warn/notify when either "native" or borg backups are not configured.
Some other stuff:
- Both Borg and borgmatic are written in Python and have associated Python packages, so it should be straightforward to integrate
- It is relatively easy to automate in ansible, but probably following the cron config for local backups, we should have a script that installs scheduled jobs from datalab directly, so there is a single source of truth.
- If this works well, we can consider massively reducing the default backup frequency for native backups, or even disabling it altogether.