offen/docker-volume-backup

"input/output error" when copying file to NFS mount

gliautard opened this issue · 4 comments

gliautard commented

What are you trying to do?
Hello 🖖,
for quite some time now I have been recieving failure email notifications from the backup containers.

Next is an example with one of them.

What is your current configuration?

services:
  jenkins:
    container_name: "${PROJECT_NAME}_jenkins"
    image: jenkins/jenkins:$JENKINS_TAG
    volumes:
      - jenkins-home:/var/jenkins_home
    labels:
      - docker-volume-backup.stop-during-backup=${PROJECT_NAME}
    restart: always
  backup:
    container_name: "${PROJECT_NAME}_backup"
    image: offen/docker-volume-backup:v2
    volumes:
      - jenkins-home:/backup/${PROJECT_NAME}/var/jenkins_home:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - nfs-backup:/archive
    env_file: ./backup.env
    restart: always

volumes:
  jenkins-home: { }
  nfs-backup:
    driver_opts:
      type: "nfs"
      o: "addr=<...>,nolock,soft,rw"
      device: ":/export/ftpbackup/<...>/cicd/${PROJECT_NAME}"
### PROJECT SETTINGS

PROJECT_NAME=jenkins
PROJECT_BASE_URL=jenkins.<...>

### --- JENKINS ----

JENKINS_TAG=lts-jdk17
BACKUP_CRON_EXPRESSION="0 2 * * *"
BACKUP_COMPRESSION="gz"
GZIP_PARALLELISM=1
BACKUP_RETENTION_DAYS="3"
NOTIFICATION_URLS="smtp://<...>:<...>@<...>:587/?fromAddress=backup@<...>&toAddresses=<...>&title=Backup notification for $PROJECT_NAME"
NOTIFICATION_LEVEL="error"
BACKUP_STOP_DURING_BACKUP_LABEL=$PROJECT_NAME
BACKUP_EXCLUDE_REGEXP="\.log$"
AWS_ENDPOINT="<...>"
AWS_ACCESS_KEY_ID="<...>"
AWS_SECRET_ACCESS_KEY="<...>"
AWS_S3_BUCKET_NAME="<...>"

Log output

Running docker-volume-backup failed with error: main.(*script).copyArchive: error copying archive: local.(*localStorage).Copy: error copying file to archive: close /archive/backup-2024-06-11T02-00-00.tar.gz: input/output error

Log output of the failed run was:

time=2024-06-11T02:00:00.642Z level=INFO msg="Stopping 1 out of 18 running container(s) as they were labeled docker-volume-backup.stop-during-backup=jenkins."
time=2024-06-11T02:02:28.766Z level=INFO msg="Created backup of `/backup` at `/tmp/backup-2024-06-11T02-00-00.tar.gz`."
time=2024-06-11T02:02:29.116Z level=INFO msg="Restarted 1 container(s)."
time=2024-06-11T02:11:08.786Z level=INFO msg="Uploaded a copy of backup `/tmp/backup-2024-06-11T02-00-00.tar.gz` to bucket `jenkins.<...>`." storage=S3
time=2024-06-11T02:28:48.643Z level=INFO msg="Removed tar file `/tmp/backup-2024-06-11T02-00-00.tar.gz`."

Additional context

  • Note that /archive is pointing on an NFS mount which is in fact a FTP (backup ftp from hosting provider) mount from another system.
  • Note that several configurations like this one run at the same time (each stack/service has it's own backup container).
m90 commented

I'm not really familiar with NFS mounts in Docker, but I would guess this is what's causing the error in your case.

Some related issues I found:

Since this tool is not concerned with mounting your volume, I am not entirely sure if such a scenario could even be fixed here. I'll leave the issue open, maybe others can chime in with their experience.

If you find out anything further about this, and there is something to be acted upon which is in the scope of this tool (even if it's just documentation), please let me know.

gliautard commented

Hello 🖖,

Thanks for your reply.

It seems related to the I/Os of that storage provided by my hosting company.

I was able to considerably reduce the number of failures by spreading the execution of the different crons over several hours by creating a delay of 20 minutes for example between each service backup.

I'm thinking of moving /archive to disk or at least something more reliable to create the archives and then maybe rsync to that FTP storage probably using something like Handle file uploads using third party tools.

m90 commented

I'm not sure about the specs of your hosting provider, but have you thought about using the SSH storage backend?

gliautard commented

Hello 🖖,

Thanks for your reply.

I can not with that specific storage so in the end I did move /archive to disk (still over NFS but not directly targeting the FTP mount), and then use rsync with the following options (in crontab) to copy the backups to the FTP mount :

#!/bin/bash

rsync -avz --no-o --no-g --stats --human-readable --delete /local-zfs-hdd/backups/stack1 /mnt/pve/ftpback-xyz1-123.provider.tld/

The problem I created for myself is now solved and I really enjoy the S3 capabilities of docker-volume-backup, it is really neat.

Thank you!