marcellodesales/sync-to-s3

Docker container that can backup files, volumes, git repos directly to Amazon S3 using s3cmd sync. It can also use a cron job to periodically upload those.

Usage

docker run -d [OPTIONS] marcellodesales/sync-to-s3 start.sh

Parameters:

-e ACCESS_KEY=<AWS_KEY>: Your AWS key.
-e SECRET_KEY=<AWS_SECRET>: Your AWS secret.
-e S3_PATH=s3://<BUCKET_NAME>/<PATH>/: S3 Bucket name and path. Should end with trailing slash.
-e GIT_REPO=git@github.company.com:org/repo-config-reference-service-config.git: GitHub repo to clone the master.
-e GIT_REV=<sha1-of-commit-of-interest>: checkout a specific commit
-v $HOME/.ssh:/root/.ssh: In case you use a git URL and require to use ssh keys.
-v /path/to/backup:/data:ro: mount target local folder to container's data folder. Content of this folder will be synced with S3 bucket.

Optional parameters:

-e PARAMS="--dry-run": parameters to pass to the sync command (full list here).
-e DATA_PATH=/data/: container's data folder. Default is /data/. Should end with trailing slash.
-e 'CRON_SCHEDULE=0 1 * * *': specifies when cron job starts (details). Default is 0 1 * * * (runs every day at 1:00 am).
no-cron: run container once and exit (no cron scheduling).

Examples:

Run upload to S3 everyday at 12:00pm:

    docker run -d \
        -e ACCESS_KEY=myawskey \
        -e SECRET_KEY=myawssecret \
        -e S3_PATH=s3://my-bucket/backup/ \
        -e 'CRON_SCHEDULE=0 12 * * *' \
        -v /home/user/data:/data:ro \
        istepanov/backup-to-s3

Run once then delete the container:

    docker run --rm \
        -e ACCESS_KEY=myawskey \
        -e SECRET_KEY=myawssecret \
        -e S3_PATH=s3://my-bucket/backup/ \
        -v /home/user/data:/data:ro \
        istepanov/backup-to-s3 no-cron

Run once to delete from s3 then delete the container:

    docker run --rm \
        -e ACCESS_KEY=myawskey \
        -e SECRET_KEY=myawssecret \
        -e S3_PATH=s3://my-bucket/backup/ \
        istepanov/backup-to-s3 delete

Run the clone for a Git Repo

docker run -ti -e ACCESS_KEY=************
               -e SECRET_KEY=***********
               -e S3_PATH=s3://springapp-config
               -e GIT_REPO=git@github.company.com:org/repo-config-reference-service-config.git
               -v $HOME/.ssh:/root/.ssh backup-s3 no-cron

Docker Compose Example

Run the clone for a Docker Volume using Docker-compose.

version: "2"

volumes:
  apks: {}

services:
  sync:
    build:
      context: .
    volumes:
      - apks:/apks
      - ./builds:/apks
    environment:
      DATA_PATH: /apks
      ACCESS_KEY: AK**************HQ
      SECRET_KEY: 4hC***************6WpYV
      S3_PATH: s3://my-s3-bucket/test/
    command: /start.sh wait-sync

Note that the local directory builds is mapped to the directory /apks from the volume apks.

$ docker-compose up
Recreating dockergitbackuptos3_sync_1
Attaching to dockergitbackuptos3_sync_1
sync_1  | Parameter is wait-sync
sync_1  | Will wait until files are at /apks...
sync_1  | Setting up watches.
sync_1  | Watches established.

At this point, you can open a new terminal and create file in the directory...

$ sudo touch builds/newfile5

Since the local dir builds/ is mapped to a docker volume, the file is then seen and all the files in the current directory is sync'ed with S3.