tornado-botocore 1.4 doesn't work with botocore-1.11.6

[Tideorz]

Currently, I'm using tornado_botocore-1.4.0 and botocore-1.11.6, both are the latest version
But when i run a simple test code

import tornado.ioloop
import tornado.web
from tornado_botocore import Botocore


class MainHandler(tornado.web.RequestHandler):
    def get(self):
        s3 = Botocore(
            service='s3',
            operation='GetObject',
            region_name='us-east-1',
            endpoint_url='http://localhost:4572')
        params = { 
            'Bucket': 'test',
            'Key': 'test.jpg',
        }   
        print s3.call(**params)


def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])  

if __name__ == "__main__":
    app = make_app()
    app.listen(54321)
    tornado.ioloop.IOLoop.current().start()

I use localstack to create a local s3 service.

export SERVICES=s3
export DEBUG=1
localstack start

and you will get Exception

Traceback (most recent call last):
  File "/data/home/china/tzhu/local/image-service/environ/lib/python2.7/site-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "app.py", line 17, in get
    print s3.call(**params)
  File "/data/home/china/tzhu/local/image-service/environ/lib/python2.7/site-packages/tornado_botocore/base.py", line 165, in call
    callback=callback
  File "/data/home/china/tzhu/local/image-service/environ/lib/python2.7/site-packages/tornado_botocore/base.py", line 121, in _make_api_call
    callback=callback
  File "/data/home/china/tzhu/local/image-service/environ/lib/python2.7/site-packages/tornado_botocore/base.py", line 106, in _make_request
    logger.debug(
AttributeError: 'Endpoint' object has no attribute 'verify'
ERROR:tornado.access:500 GET / (172.20.4.38) 2059.18ms

There are some compatible issue.
And i checked the tornado_botocore code.
https://github.com/nanvel/tornado-botocore/blob/master/tornado_botocore/base.py#L105
It will raise Exception('Endpoint' object has no attribute 'verify'.) when try to log debug.
After stepping into code, I found tornado_botocore will finally use botocore's Endpoint class,
https://github.com/boto/botocore/blob/develop/botocore/endpoint.py#L73
And you can see that the Endpoint class doesn't have the verify() method. which makes test broken.

Can someone please take a look this problem. Thanks

[dnk8n]

I am experiencing this too when attempting to use thumbor with the thumbor community aws plugin. Relevant libraries installed:

botocore==1.11.6
libthumbor==1.3.2
tc-aws==6.2.10
thumbor==6.5.2
thumbor-plugins==0.2.1
tornado==5.1
tornado-botocore==1.4.0

[mandatoryprogrammer]

Yeah I tried a bunch of monkey patching to get this library to work anywhere but had no luck. Can a disclaimer in the README be put in so people don't waste a bunch of time trying to debug this (e.g. this library doesn't work anymore)

[nanvel]

@mandatoryprogrammer good point! Mentioned it doesn't work with 1.11 in the readme.

[nanvel]

For botocore-1.12.4 + aiohttp works for me:

import json
from types import MethodType

import aiohttp

import botocore.session


class BotocoreRequest(Exception):

    def __init__(self, request, *args, **kwargs):
        super(BotocoreRequest, self).__init__(*args, **kwargs)
        self.method = request.method
        self.url = request.url
        self.headers = dict(request.headers)
        self.body = request.body and request.body.read()


def _send_request(self, request_dict, operation_model):
    request = self.create_request(request_dict, operation_model)
    raise BotocoreRequest(request=request)


class AIOAWSClient:

    def __init__(self, session, service, region, timeout=30):
        """
        session: botocore.session.get_session()
        """
        self.client = session.create_client(service, region_name=region)
        endpoint = self.client._endpoint
        endpoint._send_request = MethodType(_send_request, endpoint)
        self.timeout = timeout

    async def request(self, method, **kwargs):
        try:
            getattr(self.client, method)(**kwargs)
        except BotocoreRequest as e:
            async with aiohttp.ClientSession() as session:
                headers = {
                    k: (v.decode('utf-8') if isinstance(v, (bytes, bytearray)) else v) for k, v in e.headers.items()
                }
                async with session.request(url=e.url, method=e.method, data=e.body, headers=headers) as resp:
                    if resp.status < 300 and e.method.lower() == 'get':
                        resp_body = json.loads(await resp.text())
                    else:
                        resp_body = None
                    return resp.status, resp_body

Just replace client to tornado.httpclient and you'll not need any library.

[mandatoryprogrammer]

@nanvel Awesome, thanks for following up!