offen/docker-volume-backup

Docker swarm error

zibellon opened this issue · 5 comments

zibellon commented

Describe the bug
OffenBackup - search labels only on One node. For example Traefik (in swarm mode) search container (deploy) labels on all nodes

To Reproduce
Steps to reproduce the behavior:

  1. I have docker swarm cluster: 1 master (MASTER_1) and 2 workers (WORKER_1, WORKER_2).
  2. Run 3 different Postgres docker on all nodes (For three different projects).
  3. Want to backup all of them.
  4. For all Postgres docker add volume: - pg-backup-data:/tmp-backup
  5. For all Postgres docker add two labels (In service section, not in deploy section):
  • "docker-volume-backup.archive-pre=/bin/sh -c 'pg_dump -U postgres postgres | gzip > /tmp-backup/pg_dump.sql.gz'"
  • "docker-volume-backup.exec-label=BackupMain"
  1. On each node (MASTER_1, WORKER_1, WORKER_2) start offen-backup docker
  2. Mount volume: - pg-backup-data:/backup/pg-backup-data:ro
  3. Add labels for each offen-backup (MASTER_1, WORKER_1, WORKER_2)
  • "docker-volume-backup.archive-post=/bin/sh -c 'backup'"
  • "docker-volume-backup.exec-label=BackupMain"
  1. Start one more offen-backup on MASTER_1 = OffenBackupMain
  2. For OffenBackupMain mount: /var/run/docker.sock:/var/run/docker.sock
  3. For OffenBackupMain add env variable: EXEC_LABEL=BackupMain
  4. Go inside OffenBackupMain - docker exec -it ...
  5. Run command: backup
  6. See: backup only ONE postgres docker - on MASTER_1. Two others - not

Expected behavior
OffenBackup docker in swarm mode should search labels on all nodes

Version (please complete the following information):

  • Image Version: v2.39.1
  • Docker Version: 25.0.5

Additional context
No additional context

m90 commented

This is a limitation of Docker: when listing containers in a multi node swarm setup, you'll only see containers from a single node. This should probably be mentioned in the documentation.

If you want to achieve what you describe, you should place your labels in the deploy section so that services will be stopped and restarted instead of containers, which shall work across nodes.

zibellon commented

I try to replace Postgres docker labels into deploy section - Not working. What I can do - to backup all my Postgres docker with OffenBackup ?

m90 commented

A setup that could work for you is:

  • for each Postgres instance, deploy a dedicated backup service using the same placement rules you use for your Postgres services
  • configure each backup service to back up the postgres instance that will be placed on the same node, and nothing else

See: backup only ONE postgres docker - on MASTER_1. Two others - not

It's important to understand that the mutliple backup services are not connected in any way. If you now trigger a backup for one of the services, either manually or by cron schedule, this will only be backing up a single postgres instance. If you want to take backups of all of them, invoke the command for all of the backup containers.

m90 commented

I'm closing this as there's no more conversation, if you require further help at some point, feel free to open a new issue.

  • 👍1
zibellon commented

@m90

For each postgres deploy dedicated backup service. Okay. BUT: My postgres instances deploy on worker nodes, before backup I need to exec script (pg_dump ....) and your service cant do it, if it deploy on worker node (In swarm mode).
This solution - not work

I spent some time and write .sh script for backup WHOLE swarm cluster, depends on labels and use your open-source project

swarm-backuper.sh

  1. GET all volumes from all nodes. Start docker dind container on each node and get volume list
  2. GET all services with label volume-list. in this label - we enter volumes, which we want to backup for this service. JOIN this volumes with volumes from step one. Result: MAP<node-name, [volume1, volume2, etc]>
  3. RUN all labels: exec-pre. Run separate container on each node, where we need to run exec-pre and run docker exec ...
  4. STOP all services with label: stop. Important: stop === scale to 0, Stop only services in replicated mode
  5. RUN offen-backup on each node with all volumes from step 2
  6. RESTORE all services with label: stop. Important: we need to use the JSON from step 4. Scale to original replicas number, before step 4
  7. RUN all labels: exec-post

How we can run it

swarm-backuper-stack.yaml