helxplatform/dug

Automatic backups

cschreep opened this issue · 5 comments

cschreep commented

We need automatic backups of our ES index to easily revert or restore data

pj-linebaugh commented

We can probably use what we manually last night as a base for this. I haven't looked closely at the ES chart but hopefully some of those mounts and such are incorporated in it. Might want to jump on this while it is fresh in our minds. We can use the backup-pvc-cronjob (part of HeLx) to keep an archive of backups easily. Or maybe just take some pieces from it.

oihawkins commented

@cbizon should this move to jira?

cbizon commented

I think will get covered by the lakefs work. @YaphetKG is that correct?

YaphetKG commented

I am thinking this might be a step further down lakefs , lakefs i am thinking will keep track of what we index, i.e the source files, once they are pushed to an elastic (or redis server) I think we might still need to do some work around backing those up , but it might depend on the speed on how fast we can index, (if things to be indexed are already in lakefs and that is fast enough to restore in the respective db, or is it faster to restore it from some sort of database dump). I think it wouldn't hurt to re-visit this once we get to that point.
I'll recreate this one on jira with some notes.
cc: @hhiles

oihawkins commented

Thank you @YaphetKG ! Closing this one since you've got it covered in Jira.