botocore multipart upload limit
ngoue opened this issue · 1 comments
ngoue commented
Our database is rather large and currently we hit an error when attempting to upload a backup to S3. S3 multipart uploads only support up to 10,000 parts, but pghoard is breaking the upload into 16,000 plus parts.
Just need to re-tool the script to increase the size of each part to keep the total count equal to or under 10,000 parts total.
ngoue commented
Looks like pghoard already does some calculations to determine upload size and our instance just isn't scaled to handle the size of our database.