Limited support for S3-compatible storage
chumaumenze opened this issue ยท 7 comments
Hello @benbjohnson, first of all, thank you for your fantastic work on this project.
I am working with S3-compatible storage called Storj. I have tried to restore a replica.
Litestream config:
$ cat /etc/litestream.yml
dbs:
- path: ${database_filename}
replicas:
- type: s3
access-key-id: ${s3_accessKeyId}
secret-access-key: ${s3_secretAccessKey}
region: ${s3_region}
endpoint: ${s3_endpoint}
bucket: ${s3_bucket}
path: "${database_filename_backup_path}"
force-path-style: ${s3_forcePathStyle}
retention: 48h
snapshot-interval: 1h
Litestream restore:
database_filename=/path/to/data.db
replica_url=s3://gateway.storjshare.io/mybucket/path/to/data.db
litestream restore -v -replica s3 -o "$database_filename" $replica_url
cannot fetch generations: cannot lookup bucket region: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I think the following lines may contribute to the problem.
litestream/cmd/litestream/main.go
Lines 457 to 480 in 749bc0d
litestream/s3/replica_client.go
Lines 696 to 738 in 749bc0d
AFAIK, forcePathStyle
should be true for non-AWS S3 storage. However, the above function will overwrite forcePathStyle
for any other S3-compatible service that is not Localhost, Backblazeb2, Filebase, Digital Ocean or Linode.
I also use Storj and reach the same problem
+1 for this
I have litestream working with Oracle Cloud S3-compatible storage. The key is to use https endpoint, like this in my config:
dbs:
- path: app.db
replicas:
- type: s3
bucket: litestream
path: appdb
endpoint: https://<namespace>.compat.objectstorage.<region>.oci.customer-oci.com
Try doing the same with Storj.
The issue is valid, though, as there shouldn't be an allow/deny list for third party providers but instead just allow the user to configure forcePathStyle in all cases. The https endpoint trick is what seems to be best workaround for now.
On IDrive e2 S3-compatible storage, this works for me. (No https:// nor s3:// in endpoint.)
dbs:
- path: /path/to/data
replicas:
- type: s3
region: region_code
endpoint: ABC.region_code.idrivee2-XX.com
bucket: bucket_name
path: /path/to/replica
litestream v0.3.13
Anyone googling for Cloudflare R2 Storage for Litestream may run into the issue I kept having, where it seems to require a region to be happy. R2 has documentation on this, and effectively says just use us-east-1
which will alias to the auto region.
My configs that are working as of today, 2024-02-02:
# R2
dbs:
- path: /home/richsmith/workspace/<projectname>/storage/development.sqlite3
replicas:
- type: s3
bucket: <projectname>-db-backup
endpoint: https://XXXXXXXXXXXXXXXXXXXX.r2.cloudflarestorage.com
region: us-east-1
access-key-id: XXXXXXXXXXXXXXXXXXXXXXXX
secret-access-key: XXXXXXXXXXXXXXXXXXXX
Hope this helps the lost googler!