This is my first program in Go language. I created it for my use case - to backup files from pods running on my home k8s cluster and then restore them using initContainer - so I don't need to use persistent storage for the cluster and can easy delete/create the cluster back if required. For example, this tool make backups of my personal HomeAssistant, Vaultwarden and NextCloud
**This tool is manupulating with files on S3 bucket. Do not use this tool without specifying proper settings or you may loose your data!**Tool to backup, restore or sync local directories or MySQL database (via dump) to (or from) AWS S3 storage
Simple tool to archive all files and subdirectories of desired local directory as one ZIP or TAR archive file and upload this file to S3 bucket, to desired folder. Tool can also make mysql dump and then archive and upload it to S3 bucket/folder. Later these files anb mysql database dump can be restored by the same tool to any other directory.
The tool can also work as a daemon and run periodical backups based on desired interval. In this mode the tool also do automatic pruning and keep only number of archives in S3 bucket (deleting old files from desired folder).
In addition can sync files to/from AWS S3 bucket (as daemon) - to work as a sidecar container and allow pods to sync data.
Set the following environment variables for this tool to work:
Mandatory variables:
- AWS_BUCKET - S3 bucket to use.
- AWS_KEY - key to use to access to the bucket
- AWS_SECRET_KEY - secret key to use to access to the bucket
- DIR_TO_BACKUP - absolute path for directory to backup (tool will backup all files and subdirectories inside it)
- DIR_TO_RESTORE - absolute path for directory to restore into
- MYSQL_PASSWORD - password to connect to MySQL database to make mysql dump (if executing tool to make MySQL dump).
- PGSQL_PASSWORD - password to connect to PostgreSQL database to make pgsql dump (if use with command backup-pgsql/restore-pgsql).
Optionally, set the following variables:
- S3_BUCKET_FOLDER - folder where to store ZIP archive. "podbackup" by default
- S3_FILE_PREFIX - ZIP archive name prefix. "podbackup" by default. Full filename will be -.zip
- S3_ENDPOINT - set URL for S3 storage other than AWS (i.e. minio S3 storage). Works only for s3 sync feature! URL format is :
- S3_SYNC_PARALLELISM - set number of parralel jobs to sync. Works only for s3 sync feature!
- S3_COPY_BEFORE_SYNC - copy all files from S3 folder before start sync-to-s3 process (useful if required to pull all files from S3 after pod creation). False by default.
- S3_COPY_WITH_DELETE - when copying to/from S3 make destination looks like source (delete files in destination if they don't exist in source)
- ENCRYPT_PASSWORD - encrypt/decrypt ZIP archives using this password.
- BACKUP_INTERVAL - interval in seconds (if number like
3000) or in minutes/hours (like2mor24h) to run periodical backups (if running as daemon). 1h by default. - PRUNE_INTERVAL - interval in seconds (if number like
3000) or in minutes/hours (like2mor24h) to run periodical pruning (if running as daemon). 2h by default. - COPIES_TO_KEEP - number of copies to keep in S3 folder when executing pruning.
- FORCE_RESTORE - set to True if needed the tool to fail (exit with code 1) if it cannot restore files from backup. Useful if tool is using as an IniContainer and you don't want main containers to run without restoring actual data.
- MYSQL_USER - user to connect to MySQL database when making mysql dump. Default value as root.
- MYSQL_HOST - IP address or hosname to use to connect to MySQL database. Default value is 127.0.0.1. Process will wait for connection to restore the database.
- MYSQL_PORT - port to use to connect to MySQL database. Default value is 3306.
- PGSQL_USER - user to connect to PostgreSQL database when making pgsql dump. Default value as root.
- PGSQL_HOST - IP address or hosname to use to connect to PostgreSQL database. Default value is 127.0.0.1. Process will wait for connection to restore the database.
- PGSQL_PORT - port to use to connect to PostgreSQL database. Default value is 5432.
- ARCHIVE_TYPE - by default set to
zip- tool will create ZIP archive (and encrypt it if ENCRYPT_PASSWORD is set). Set totarzip- to archive all files as tar archive and then zip it (encrypted if ENCRYPT_PASSWORD is set). Use it if you need to save original ownership and mode of the files. Set totargz- to archive all files as tar compressed archive. File mode and ownership persist during unpacking, however encryption is not supported.
Run
podbackup <command>
where commands are:
-
backup- run one time backup and backup all files from folderDIR_TO_BACKUPto S3 object storage. -
backup-sql- run one time backup to make MySQL dump based onMYSQL*variables. -
backup-pgsql- run one time backup to make PostgreSQL dump based onPGSQL*variables. -
backup-daemon- work as daemon and run periodical backups according to BACKUP_INTERVAL environment variable (1hby default). In this mode daemon will do automatic pruning (default PRUNE_INTERVAL is2h) and keep only # of copies based on COPIES_TO_KEEP environment variable (3 by default). All other files with prefixS3_FILE_PREFIXin the folderS3_BUCKET_FOLDERwill be destroyed. -
backup-sql-daemon- work as daemon and run periodical MySQL database dumps according to BACKUP_INTERVAL environment variable (1hby default). In this mode daemon will do automatic pruning (default PRUNE_INTERVAL is2h) and keep only # of copies based on COPIES_TO_KEEP environment variable (3 by default). All other files with prefixS3_FILE_PREFIXin the folderS3_BUCKET_FOLDERwill be destroyed. -
backup-pgsql-daemon- work as daemon and run periodical PostgreSQL database dumps according to BACKUP_INTERVAL environment variable (1hby default). In this mode daemon will do automatic pruning (default PRUNE_INTERVAL is2h) and keep only # of copies based on COPIES_TO_KEEP environment variable (3 by default). All other files with prefixS3_FILE_PREFIXin the folderS3_BUCKET_FOLDERwill be destroyed. -
prune- manually run pruning (delete all old archives). All other files with prefixS3_FILE_PREFIXin the folderS3_BUCKET_FOLDERwill be destroyed. -
list- list files in S3 folder (based onS3_BUCKET_FOLDERenvironment variable). -
restore- download file from S3 and restore files to directory (DIR_TO_RESTORE environment variable). Most recent archive will be used. To restore from another file provide archive name based on 'podbackup list' output (like podbackup/podbackup-20210802213807.zip) as an argument forrestorecommand. -
restore-sql- download file from S3 and restore MySQL database. Most recent archive will be used. To restore from another file provide archive name based on 'podbackup list' output (like podbackup/podbackup-20210802213807.zip) as an argument forrestore-sqlcommand. -
copy-to-s3- copy content of local folder DIR_TO_BACKUP into S3 storage, to AWS_BUCKET and S3_BUCKET_FOLDER. -
copy-from-s3- copy content from S3 storage, AWS_BUCKET and S3_BUCKET_FOLDER to local folder DIR_TO_RESTORE. -
sync-to-s3- sync content of local folderDIR_TO_BACKUPinto S3 storage, toAWS_BUCKETandS3_BUCKET_FOLDER. Works as daemon and runs sync process periodically according toBACKUP_INTERVALenvironment variable. If file exists on S3 bucket the tool will skip copying it. -
sync-from-s3- sync content from S3 storage,AWS_BUCKETandS3_BUCKET_FOLDERto local folderDIR_TO_RESTORE. Works as daemon and runs sync process periodically according toBACKUP_INTERVALenvironment variable. If file exists on local filesystem the tool will skip copying it.
I'm using this tool to backup/restore my Home Assistant, Vaultwarden, Trilium and NextCloud data - so I don't need to use any localstorage in my home k8s cluster.
The tool is working as a sidecar container for home-assistant pod and runs periodical backups (once per hour). To restore the data the Home Assistant k8s deployment has initContainer with the same tool performing restore process.
Example of VaultWarden deployment
Example of MySQL deployment
Example of NextCloud deployment
Example of home-assistant k8s deployment below or here
apiVersion: apps/v1
kind: Deployment
metadata:
name: home-assistant
namespace: home-assistant
labels:
app.kubernetes.io/name: home-assistant
spec:
revisionHistoryLimit: 3
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: home-assistant
app.kubernetes.io/instance: home-assistant
template:
metadata:
labels:
app.kubernetes.io/name: home-assistant
app.kubernetes.io/instance: home-assistant
spec:
serviceAccountName: default
terminationGracePeriodSeconds: 90
dnsPolicy: ClusterFirst
enableServiceLinks: true
volumes:
- name: hass-workingdir
emptyDir: {}
containers:
- name: home-assistant
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
image: "homeassistant/home-assistant:2021.6.3"
imagePullPolicy: IfNotPresent
securityContext:
privileged: false
env:
- name: "TZ"
value: "UTC"
ports:
- name: http
containerPort: 8123
protocol: TCP
volumeMounts:
- name: hass-workingdir
mountPath: /config
livenessProbe:
tcpSocket:
port: 8123
initialDelaySeconds: 30
failureThreshold: 10
timeoutSeconds: 1
periodSeconds: 10
readinessProbe:
tcpSocket:
port: 8123
initialDelaySeconds: 30
failureThreshold: 10
timeoutSeconds: 1
periodSeconds: 10
startupProbe:
tcpSocket:
port: 8123
initialDelaySeconds: 300
failureThreshold: 30
timeoutSeconds: 2
periodSeconds: 5
- name: hass-backup-daemon
image: quay.io/vadimzharov/podbackup:latest
lifecycle:
preStop:
exec:
command: ["/usr/local/bin/podbackup", "backup"]
command: ["podbackup"]
args: ["backup-daemon"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 50m
memory: 64Mi
volumeMounts:
- name: hass-workingdir
mountPath: /hass-workdir
env:
- name: AWS_BUCKET
valueFrom:
secretKeyRef:
name: backup-restore-creds
key: awsbucket
- name: AWS_KEY
valueFrom:
secretKeyRef:
name: backup-restore-creds
key: awskey
- name: AWS_SECRET_KEY
valueFrom:
secretKeyRef:
name: backup-restore-creds
key: awssecretkey
- name: S3_BUCKET_FOLDER
value: "homehass"
- name: S3_FILE_PREFIX
value: "hass-backup"
- name: DIR_TO_BACKUP
value: "/hass-workdir"
- name: BACKUP_INTERVAL
value: "1h"
initContainers:
- name: hass-restore
image: quay.io/vadimzharov/podbackup:latest
command: ["podbackup"]
args: ["restore"]
volumeMounts:
- name: hass-workingdir
mountPath: /hass-workdir
env:
- name: AWS_BUCKET
valueFrom:
secretKeyRef:
name: backup-restore-creds
key: awsbucket
- name: AWS_KEY
valueFrom:
secretKeyRef:
name: backup-restore-creds
key: awskey
- name: AWS_SECRET_KEY
valueFrom:
secretKeyRef:
name: backup-restore-creds
key: awssecretkey
- name: S3_BUCKET_FOLDER
value: "homehass"
- name: S3_FILE_PREFIX
value: "hass-backup"
- name: DIR_TO_RESTORE
value: "/hass-workdir"