@tus/s3-store: add option to disable object tags
adrianocr opened this issue · 10 comments
I was using uppy with tusd and the S3 datastore -- which works great. But my project is pretty much 90% JS so I wanted to reuse some stuff like my JS database client. This led me to give tus-node-server a try. With @tus/file-store it works fine, but with @tus/s3-store uppy fails to upload the file with the following error: Something went wrong with that request Invalid URL
I'm using pretty much the exact sample from the s3Store package's readme:
const {Server, EVENTS} = require("@tus/server");
const {S3Store} = require("@tus/s3-store");
// this works correctly and logs the same correct values that work perfectly with tusd
console.log(process.env.S3_BUCKET, process.env.S3_REGION, process.env.S3_ENDPOINT, process.env.S3_ACCESS_KEY_ID, process.env.S3_SECRET_ACCESS_KEY);
const s3Store = new S3Store({
s3ClientConfig: {
bucket: process.env.S3_BUCKET,
region: process.env.S3_REGION,
endpoint: process.env.S3_ENDPOINT,
credentials: {
accessKeyId: process.env.S3_ACCESS_KEY_ID,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
},
},
});
const tusServer = new Server({
path: "/files",
datastore: s3Store,
});
console.log("Listening on :3000");
tusServer.listen({host: '0.0.0.0', port: 3000});
My uppy config is:
new Uppy({meta: {shop: shop}}).use(Tus, {
endpoint: 'https://node-tus-xxxxxxxxxx.up.railway.app/files/',
}),
can you try using the onResponseError function on the Server to inspect the error you are getting, that might help know the cause of this error
const tusServer = new Server({
path: "/files",
datastore: s3Store,
onResponseError: (req, res, err) => {
console.error(err)
}
});This is the error log:
onResponseError
InvalidArgument: Unsupported header 'x-amz-tagging' received for this API call.
at throwDefaultError (/app/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
at /app/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
at de_PutObjectCommandError (/app/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4841:12)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async /app/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
at async /app/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:30:20
at async /app/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:31:46
at async /app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:63:20
at async /app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
at async /app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20 {
'$fault': 'client',
'$metadata': {
httpStatusCode: 400,
requestId: '1289c689d96015bc',
extendedRequestId: 'adcNu/WuvbnZvUHdPbpw=',
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
Code: 'InvalidArgument'
}
I did a little googling and it seems that backblaze b2 (which is my storage provider) has issues with the x-amz-tagging header: https://www.backblaze.com/docs/cloud-storage-s3-compatible-api#get-object-tagging
Any way to remove that header from the COPY and PUT object calls?
mmmh, I see - it seems that this issue is caused by the expiration extension which uses tags.
I'm wondering if we should disable tagging if the expiration extension is not used
https://github.com/tus/tus-node-server/blob/main/packages/s3-store/index.ts#L107
We don't provide a way to turn off tagging, currently, we tested this on AWS S3 / Minio which both supports tagging hence we didn't realise that tagging is not always implemented in all stores
Probably is something we should provide an option in the s3 store
something like:
const s3Store = new S3Store({
useTags: false, // <-- new option
s3ClientConfig: {
...
},
});alternatively we might to do the otherway around where we have to specify a tag name for opting-in on object tags:
const s3Store = new S3Store({
tagName: 'Tus-Completed', // <-- new option
s3ClientConfig: {
...
},
});Thoughts @Murderlon
In the meantime I migrated to using wasabi which just ignores the x-amz-tagging header. Out of curiosity I was also considering using DigitalOcean Spaces (S3 compatible) and they also ignore the x-amz-tagging header. So it seems only AWS S3 and perhaps a few others actually make use of that tag.
I believe your suggested approach where the tag can be disabled or opt-in would be ideal.
if they ignore it and do not throw an error is fine, i was surprised that backblaze b2 errors out
Indeed it's a bit odd that Backblaze throws an error. But an option to disable tags makes sense. The boolean options seems more logical for this use case 👍
Wanted too.
Nothing will break if I don't use extensions for expiration and do this patch?
diff --git a/dist/index.js b/dist/index.js
index cd35efbf3298386fa77a75bae10f43ac92e8b4c8..8e72d9e1369fdac7d3616824a1ee1a816f9296d9 100644
--- a/dist/index.js
+++ b/dist/index.js
@@ -106,7 +106,6 @@ class S3Store extends server_1.DataStore {
Bucket: this.bucket,
Key: this.infoKey(upload.id),
Body: JSON.stringify(upload),
- Tagging: `Tus-Completed=false`,
Metadata: {
'upload-id': uploadId,
'tus-version': server_2.TUS_RESUMABLE,
@@ -120,7 +119,6 @@ class S3Store extends server_1.DataStore {
Bucket: this.bucket,
Key: this.infoKey(upload.id),
Body: JSON.stringify(upload),
- Tagging: `Tus-Completed=true`,
Metadata: {
'upload-id': uploadId,
'tus-version': server_2.TUS_RESUMABLE,
@@ -184,7 +182,6 @@ class S3Store extends server_1.DataStore {
Bucket: this.bucket,
Key: this.partKey(id, true),
Body: readStream,
- Tagging: 'Tus-Completed=false',
});
log(`[${id}] finished uploading incomplete part`);
return data.ETag;@rougsig no, nothing will break, since tagging is only a useful feature for cleaning up assets using lifecycle rules
Will raise a PR sometime soon to address this
PR is up 👆