S3QL
S3QL is a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access. http://www.rath.org/s3ql-docs/about.html
This project aims to install S3QL in a Docker container and use it in a stack to store data on the cloud. Since bind propagation is not configurable for volumes, it can only be used through bind mounts.
Warning
This project is experimental and not fully tested. Use it at your own risk.
So far, it has been tested with the following storage backends:
- OpenStack Swift (used in the CI/CD testing stage)
- DigitalOcean Spaces
If you encounter any issues with other storage backends, feel free to submit an Issue or, preferably, directly file a Merge Request.
Requirements
fuse
must be installed on the host.
Debian
apt-get install fuse
Build
S3QL version must be privided in variable S3QL_VERSION
for the image to be build.
docker build -t docker-s3ql --build-arg S3QL_VERSION=3.3.2 .
Usage
Environment variables
S3QL_USERNAME
andS3QL_PASSWORD
: Cloud service credentials.S3QL_PROJECT
: Cloud project or tenant (OpenStack/Swift Backends only).S3QL_URL
: Depends on the backend. See S3QL backends documentation.FS_PASSPHRASE
: S3QL FS may be encrypted. This is the passphrase to unlock the AES 256 encryption key.S3QL_MOUNT_OPTIONS
: Options be added to themount.s3ql
command in addition to--fg
. Seemount.s3ql
documentation.S3QL_FSCK_OPTIONS
: Options to be added to thefsck.s3ql
command in addition to--batch
. Seefsck.s3ql
documentation .
Mount a S3QL FS
docker pull registry.gitlab.com/salokyn/docker-s3ql:latest
docker run -d -e S3QL_USERNAME=myLogin \
-e S3QL_PASSWORD=myPassword \
-e S3QL_PROJECT=myTenant \
-e S3QL_URL=swiftks://openstack.backend/REGION:CONTAINER \
-e FS_PASSPHRASE=mySecretPassphrase \
-v /s3ql:/s3ql:rw,rshared \
--cap-add SYS_ADMIN \
--device /dev/fuse \
--security-opt apparmor:unconfined \
--stop-timeout 10m \
registry.gitlab.com/salokyn/docker-s3ql:latest
You can also use a docker-compose.yml
file :
version: '3.5'
services:
s3ql:
image: registry.gitlab.com/salokyn/docker-s3ql
volumes:
- /bind/path/on/host:/s3ql:shared
environment:
- S3QL_PASSWORD=myPassword
- S3QL_USERNAME=myLogin
- S3QL_PROJECT=myTenant
- S3QL_URL=swiftks://openstack.backend/REGION:CONTAINER
- FS_PASSPHRASE=mySecretPassphrase
security_opt:
- apparmor:unconfined
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse
stop_grace_period: 10m
app:
...
depends_on:
- s3ql
volumes:
- /bind/path/on/host:/s3ql:slave
...
Take care to use bind-propagation to share S3QL mount with the host or another container. You should specify a stop-timout greater than default 10s since unmounting S3QL may be long.
Create a S3QL FS
docker pull registry.gitlab.com/salokyn/docker-s3ql:latest
docker run -ti -e S3QL_USERNAME=myLogin \
-e S3QL_PASSWORD=myPassword \
-e S3QL_PROJECT=myTenant \
-e S3QL_URL=swiftks://openstack.backend/REGION:CONTAINER \
registry.gitlab.com/salokyn/docker-s3ql:latest \
mkfs.s3ql swiftks://openstack.backend/REGION:CONTAINER