Space required calculation
xiano8494 opened this issue · 8 comments
I'm not sure how accurate the estimated space required is. Perhaps there is a requirement for a large amount of space whilst completing the backups, but this is going to cause issues for me potentially.
Looking at the messages (pasted below) my last backup took up 53GB, but the estimate for a backup is 148GB! I assume that it is calculated by looking at /var/www and then using a 3x multiple. This is likely a little high? The way I see it, we take a copy of that location, and we then create an archive of the location, so worst case it is going to 2x /var/www + whatever the DBs take up. Perhaps for smaller sites 3x /var/www works, but for larger sites, it doesn't make sense.
"You have 1 local backups occupying 53017 MB of space.
Estimated backup size [148188 MB] LARGER than available disk space [138618 MB]."
That's correct. The estimation is /var/www/ x2 if S3 is not enabled as all files are copied into one single dir [1x] + they're archived [1x (actually 0.6x in case of bz2 but we consider it a 1x anyway)], and x3 in case of S3 since we need to restore the entire archive with duplicity from S3 to disk for integrity purposes before we upload with duplicity. This is independent of backup_local_storage_location which we've implemented the other day, as all calculations are in addition to the actual /var/www/ dir.
Just for my understanding, could you clarify what you mean by "x3 in case of S3 since we need to restore the entire archive with duplicity from S3 to disk for integrity purposes before we upload with duplicity"
In addition to the copy and the archive, we also fully download the last backup stored on S3 to verify that we can decrypt it correctly [therefore occupying another 1x of space] before we actually upload anything new. This is because duplicity will not warn in case of a new, different key being used if that were to happen by mistake. I guess we could perform the test restore before we actually backup anything locally, let me reopen this to look into it.
Simply moving the test restore a few lines up, purging it, and only then backing up seems to work, let me do some additional testing and I'll push a beta.
Tested, works for me, please try a re-install in beta and let me know. I haven't updated the space required calculation yet, let's first see if the backup script works with this order of operations. Thanks
@xiano8494 any news? Thanks for your help.
I'll test to see if I have any issues with the amendments. Although, of course, my S3 backups aren't completing at the moment anyway!
Ok, your comment on #17 actually proves this is working, so I'm closing this for now. Thanks.