ExpiredToken error when `cp` large dataset
MqllR opened this issue · 0 comments
MqllR commented
Some context:
- large dataset: > 4TB, million of files
- s5cmd running in AWS EKS with assumeRole
I'm facing intermittent problem to upload or download with the following error:
MultipartUpload: upload multipart failed upload id: xxx-- caused by: ExpiredToken: The provided token has expired. status code: 400
After looking at the code, there is an explicit deny for retrying this kind of error.
The PR introducing this code seems to tackle another kind of issue. So I'm wondering if that would make sense to revisit the s3 session management.