nanvel/tornado-botocore

Improper Content Length Checking for gzip responses

TelegramSam opened this issue · 0 comments

I've run into an error where a gzipped response fails the content_length check that is part of the botocore.response.StreamingBody .read() method. The content_length reported is the header returned by the s3 get call, but the buffer has at this point been already decompressed by the tornado http client.

I'm not sure the best way to solve this. I've used the following modification, but I'm unsure this is the cleanest way to solve this:

base.py starting line 69:

   elif operation_model.has_streaming_output:
       content_length = response_dict['headers'].get('content-length')
       if response_dict['headers'].get('X-Consumed-Content-Encoding') == 'gzip':
           content_length = None
           # this prevents from checking the decompressed stream length against the compressed content-length
       response_dict['body'] = botocore.response.StreamingBody(
           http_response.buffer, content_length)

This detects the decompression effect, and sends in a content_length of None to bypass the content length checking within the botocore library.

If somebody can provide me some feedback, I'd be happy to submit a pull request with the change.