rclone multipart upload starts to fail after some time

[artur9010]

Current Behavior

2024/06/07 16:40:55 DEBUG : file: Received error: multi-thread copy: failed to close object after copy: failed to complete multipart upload "65ba3589a8216c1299d3baa23d429a8f726521c1d6107ec10711e686fa8c5e2e": InternalError: failed to complete multipart upload: transaction failed (attempt 1): failed to complete multipart upload: failed to fetch multipart upload: object id mismatch: /bucket1/otherfile != /bucket2/file: object not found

	status code: 500, request id: 1649C7747B70E1C0, host id: MTY0OUM3NzQ3QjcwRTFDMDE2NDlDNzc0N0I3MEUxQzAxNjQ5Qzc3NDdCNzBFMUMwMTY0OUM3NzQ3QjcwRTFDMA

In renterd worker logs I saw only this

2024-06-07T14:40:54Z	ERROR	s3.s3	InternalError: failed to complete multipart upload: transaction failed (attempt 1): failed to complete multipart upload: failed to fetch multipart upload: object id mismatch: /bucket1/otherfile != /bucket2/file: object not found

(file names removed, those two objects have different names and buckets)

After some time (13 minutes) of running rclone sync multipart uploads starts to fail due to object id mismatch, I also saw in ui there are some unfinished multipart uploads from yesterday. I removed all of those uploads, started rclone sync again and currently (7 minutes after start) there are no problems.

Used flags in sync:

-vv --progress --s3-chunk-size 20MiB --transfers 8 

Expected Behavior

Well... not fail

Steps to Reproduce

No response

Version

80b7959

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

Ubuntu 22.04, kubernetes

Autopilot Config

{ "contracts": { "set": "autopilot", "amount": 50, "allowance": "70000000000000000000000000000", "period": 6048, "renewWindow": 2016, "download": 4200000000000, "upload": 7000000000000, "storage": 3000000000000, "prune": true }, "hosts": { "allowRedundantIPs": false, "maxDowntimeHours": 336, "minProtocolVersion": "1.6.0", "minRecentScanFailures": 10, "scoreOverrides": null } }

Bus Config

{ "default": "autopilot" } { "hostBlockHeightLeeway": 6, "maxContractPrice": "15000000000000000000000000", "maxDownloadPrice": "1544295232000000000000000000", "maxRPCPrice": "1000000000000000000000", "maxStoragePrice": "350743725231", "maxUploadPrice": "134977509000000000000000000", "migrationSurchargeMultiplier": 10, "minAccountExpiry": 86400000000000, "minMaxEphemeralAccountBalance": "1000000000000000000000000", "minPriceTableValidity": 300000000000 } { "minShards": 5, "totalShards": 15 } { "enabled": true, "slabBufferMaxSizeSoft": 4294967296 }

Contract Set Contracts

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 16676 0 16676 0 0 129k 0 --:--:-- --:--:-- --:--:-- 130k 19

Anything else?

No response