Mem limiter affecting get throughput in corner cases

[DmitriyMusatkin]

Describe the bug

Mem limiter provides a push back mechanism on scheduler if memory usage is getting close to the limit.

With gets there is a chicken and egg problem, since we dont know the size object before doing a get and we want to avoid making additional request to figure out that size before doing a get (cause additional roundtrips for get tank perf). So crt will optimistically do a ranged get with a part size to get a first ranged part and figure out the overall size.

This approach works fine in most cases. But it will unnecessarily slow down gets when part size is huge and gets itself are small. Ex. part size is 1 GB and the files being retrieved are 1mb. Mem limiter in that case would only be able to schedule 4 gets in parallel (assuming 4 gb mem limit), since it would account for the worst case of getting back 1GB part. But in practice we should be able to schedule a lot more gets in parallel, cause they are all small.

refer to aws/aws-sdk-cpp#2922 for example of this in the wild

Expected Behavior

something better?

Current Behavior

download slows down to a crawl on lots of small gets if part size is huge

Reproduction Steps

set part size to a gig and observe downloads on 10k 256kb files

Possible Solution

No response

Additional Information/Context

No response

aws-c-s3 version used

latest

Compiler and version used

every compiler

Operating System and version

every os