[s3] $INDEX in s3_key_format can be incorrect with multipart uploads
Opened this issue · 4 comments
Am I impacted?
If you have an S3 output: https://docs.fluentbit.io/manual/pipeline/outputs/s3
And have use_put_object false (the default) and have an s3_key_format that uses $INDEX you may be impacted.
Bug Impact
When a multipart UploadPart call fails, the index can be incorrectly decremented, leading it to be wrong (smaller value that expected) for the next file upload. This may cause the same INDEX number to be used repeatedly in a row.
Root Cause: Code Bug
The code should only decrement the $INDEX when the multipart upload is marked for failure.
Currently it decrements here instead:
Quick fix for AWS distro: https://github.com/PettitWesley/fluent-bit/pull/25/files
The S3 stability refactor that I am working on here also has a full logic fix: https://github.com/PettitWesley/fluent-bit/pull/24/files
This will be fixed once I get 2.31.12 out: https://github.com/aws/aws-for-fluent-bit/blob/mainline/CHANGELOG.md