Error restarting containers after backup
andrewmooreio opened this issue · 4 comments
What are you trying to do?
I've been using this container for months to back up my other containers to S3. However, in the past 2 days, the backup succeeds but then the containers which are stopped during the backup (docker-volume-backup.stop-during-backup=true) fail to start again.
The logs indicate a log of storage space for the /var/lib/docker directory, however there is more than enough, and I can start the containers manually without any issue at all.
Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 2.3M 1.6G 1% /run
/dev/mapper/docker01--vg-root 14G 3.2G 9.9G 25% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda2 456M 106M 325M 25% /boot
/dev/mapper/docker--vg-docker--pool 126G 70G 50G 59% /var/lib/docker
/dev/sda1 511M 5.9M 506M 2% /boot/efi
tmpfs 1.6G 0 1.6G 0% /run/user/1000
What is your current configuration?
backup:
container_name: backup
environment:
- AWS_ENDPOINT=${S3_ENDPOINT}
- AWS_S3_BUCKET_NAME=${S3_BUCKET_NAME}
- AWS_ACCESS_KEY_ID=${S3_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY}
- AWS_S3_PATH=${HOSTNAME_OVERRIDE}
- BACKUP_CRON_EXPRESSION=0 5 * * *
- BACKUP_RETENTION_DAYS=14
- NOTIFICATION_URLS=${BACKUP_NOTIFICATION_URLS}
image: offen/docker-volume-backup:latest
network_mode: bridge
restart: always
volumes:
- etc...
Log output
Running docker-volume-backup failed with error: stopContainers: 12 error(s) restarting containers and services: Error response from daemon: mkdir /var/lib/docker/overlay2/387d1fc4adc47bea32c4cbe6b8d1801f934095effdc043c193578413c0a0af7f/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/d9c62ed8d24cd03314635c884f233fd22ea35e56e98d25b8c4a619cf3270b52d/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/c2647ae33303bf6d907365d27f267aaaff67df5e1e66c16f41fb33925a1d08c6/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/e5f273e5a23d2127dee0cca12d2e17e9416a4c660c1b3106e61654842a1e300a/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/ecc9bfcccc23d8a7b919cc6a5573e9d893d4fccad8fe2f86f7d092e276bb17aa/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/097c683d6ff5f374e8a5c94068ea9fba3d8b9ff9dcbe87638079134a8807e0a3/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/f98c0782b746246ecc0d8146b909582373a7a1d676e65862609f5975d7d8ac68/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/97e6990e4ed4b42c6430b284b0bf02f32b3716d0e34a261430a725aead0fed51/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/3484c02aff3d8e5891dde90d25eaa8505ae35730b7fd9ac2002f715e64e0120d/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/90b7482baa0932f7442576d946a8e7a75521d3ae71dd2a002bc255f4e106ac9e/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/d78788b70e9a1eb0fedc315ad2f5e847ba086cef3940f3a4d0c3c29c98400238/merged: no space left on device
Error response from daemon: mkdir /var/lib/docker/overlay2/c3f89baa3a47a65baf6d44da33842a945caad3440040910079c6b03856c00a20/merged: no space left on device
Additional context
Thanks in advance!
I also ran into this issue in the past. In my case it was not running out of disk space, but running out of available inodes on the ext4 filesystem. You can check this by running df -i.
The solution for me was to configure log rotation on the json-file ( default ) logger. Here is the related article from Docker.
Thanks, @pixxon. I hadn't thought of inodes. However it seems I have plenty free.
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 2040899 415 2040484 1% /dev
tmpfs 2046874 1213 2045661 1% /run
/dev/mapper/docker01--vg-root 920272 49117 871155 6% /
tmpfs 2046874 1 2046873 1% /dev/shm
tmpfs 2046874 3 2046871 1% /run/lock
/dev/sda2 124928 347 124581 1% /boot
/dev/mapper/docker--vg-docker--pool 8388608 598435 7790173 8% /var/lib/docker
/dev/sda1 0 0 0 - /boot/efi
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/9e551cecdca009b0f6e1bc893496eec414d09d790385d28cc3fdc9ebeb4a0d94/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/c3f89baa3a47a65baf6d44da33842a945caad3440040910079c6b03856c00a20/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/8263f2a866454156ae772f1ce3c83655e1328330356c35e5bad6cacc2a1cdaef/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/c7974e6c3347f36b336cebe3b385d518fa801ac70aec5a18c2ed0f50e2b2115f/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/90b7482baa0932f7442576d946a8e7a75521d3ae71dd2a002bc255f4e106ac9e/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/a12410dedd580814d67c5884812e4db3fb09f2366733db2968572ddbd0815761/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/d78788b70e9a1eb0fedc315ad2f5e847ba086cef3940f3a4d0c3c29c98400238/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/097c683d6ff5f374e8a5c94068ea9fba3d8b9ff9dcbe87638079134a8807e0a3/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/387d1fc4adc47bea32c4cbe6b8d1801f934095effdc043c193578413c0a0af7f/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/3484c02aff3d8e5891dde90d25eaa8505ae35730b7fd9ac2002f715e64e0120d/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/d9c62ed8d24cd03314635c884f233fd22ea35e56e98d25b8c4a619cf3270b52d/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/c2647ae33303bf6d907365d27f267aaaff67df5e1e66c16f41fb33925a1d08c6/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/e5f273e5a23d2127dee0cca12d2e17e9416a4c660c1b3106e61654842a1e300a/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/ecc9bfcccc23d8a7b919cc6a5573e9d893d4fccad8fe2f86f7d092e276bb17aa/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/e75cb9857c417d4d1500aa5fee7ec9203e981b0d0b155625ca7757255244c2bb/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/1b48d4efb2267947703a61e125d8de590a5d8edb28b8818328f0473a31176c0d/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/f98c0782b746246ecc0d8146b909582373a7a1d676e65862609f5975d7d8ac68/merged
overlay 8388608 598435 7790173 8% /var/lib/docker/overlay2/97e6990e4ed4b42c6430b284b0bf02f32b3716d0e34a261430a725aead0fed51/merged
tmpfs 409374 13 409361 1% /run/user/1000
Unless this started happening after you did an image upgrade (which I think you did not?), I'm not sure if there is anything actionable regarding this tool here. The docker daemon on your setup fails to do something and the error is being forwarded.
It is indeed surprising seeing that your disk seems to have enough space and that you can restart containers manually, but I fail to see how the stop/restart mechanism in this image could be related.
One thing I am wondering: are the archives you are creating rather large? I.e. could storing them temporarily while uploading them to S3 exhaust (or almost exhaust) your available disk space (i.e. #95 and #316)?
Unless this started happening after you did an image upgrade (which I think you did not?), I'm not sure if there is anything actionable regarding this tool here. The docker daemon on your setup fails to do something and the error is being forwarded.
It is indeed surprising seeing that your disk seems to have enough space and that you can restart containers manually, but I fail to see how the stop/restart mechanism in this image could be related.
One thing I am wondering: are the archives you are creating rather large? I.e. could storing them temporarily while uploading them to S3 exhaust (or almost exhaust) your available disk space (i.e. #95 and #316)?
That was it, everything looks fine storage wise the day after, but of course by then the backup archive has been deleted.
Thanks for the help!