aiohttp mocking binary content

[sky-code]

I am refactoring one of my project from using requests to aiohttp as http client library and my exists test failing.
I found 2 issues in mockig aiohttp
First is url handling, when i write test for requests library - pook require full url to match

url = 'http://data.alexa.com/data?cli=10&dat=snbamz&url=http%3A%2F%2Ftest.org'

when i replace requests to aiohttp pook stop matching full url and need now url without parameters

url = 'http://data.alexa.com/data'

Second issue with content mocking, when i use requests i mock binary content

    mock_response_content = b'<?xml version="1.0" encoding="UTF-8"?>\r\n\r\n<!-- Need more Alexa data?  Find our APIs here: https://aws.amazon.com/alexa/ -->\r\n<ALEXA VER="0.9" URL="test.org/" HOME="0" AID="=" IDN="test.org/">...</ALEXA>'

this code now throw exception with aiohttp but not throw any exception with requests
here is exception

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
natrix\plugins\alexa_data\alexa_data.py:20: in perform_scan
    async with self.http_session.get(DATA_ALEXA_API_URL, params=alexa_api_params) as resp:
env\lib\site-packages\aiohttp\client.py:529: in __aenter__
    self._resp = yield from self._coro
env\lib\site-packages\pook\interceptors\aiohttp.py:124: in handler
    data=data, headers=headers, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pook.interceptors.aiohttp.AIOHTTPInterceptor object at 0x051331D0>
_request = <function ClientSession._request at 0x04578B28>
session = <aiohttp.client.ClientSession object at 0x04E968D0>, method = 'GET'
url = 'http://data.alexa.com/data', data = None, headers = []
kw = {'allow_redirects': True, 'params': {'cli': '10', 'dat': 'snbamz', 'url': 'http://test.org'}}
req = Request(
  method=GET,
  headers=HTTPHeaderDict({}),
  body=None,
  url=http://data.alexa.com/data,
  query={},
)
mock = Mock(
  matches=1,
  times=0,
  persist=False,
  matchers=MatcherEngine([
    MethodMatcher(GET),
    URLMatcher(http:...ARITY URL="test.org/" TEXT="2285614" SOURCE="panel"/><REACH RANK="1897255"/><RANK DELTA="+608439"/></SD></ALEXA>'
  )
)
res = Response(
    headers=HTTPHeaderDict({}),
    status=200,
    body=b'<?xml version="1.0" encoding="UTF-8"?>\r\n\r\n<!-...OPULARITY URL="test.org/" TEXT="2285614" SOURCE="panel"/><REACH RANK="1897255"/><RANK DELTA="+608439"/></SD></ALEXA>'
)
_res = <ClientResponse(http://data.alexa.com/data) [200 OK]>
<CIMultiDictProxy()>


    @asyncio.coroutine
    def _on_request(self, _request, session, method, url,
                    data=None, headers=None, **kw):
        # Create request contract based on incoming params
        req = Request(method)
        req.headers = headers or {}
        req.body = data
    
        # Expose extra variadic arguments
        req.extra = kw
    
        # Compose URL
        req.url = str(url)
    
        # Match the request against the registered mocks in pook
        mock = self.engine.match(req)
    
        # If cannot match any mock, run real HTTP request if networking
        # or silent model are enabled, otherwise this statement won't
        # be reached (an exception will be raised before).
        if not mock:
            return _request(session, method, url,
                            data=data, headers=headers, **kw)
    
        # Simulate network delay
        if mock._delay:
            yield from asyncio.sleep(mock._delay / 1000)  # noqa
    
        # Shortcut to mock response
        res = mock._response
    
        # Aggregate headers as list of tuples for interface compatibility
        headers = []
        for key in res._headers:
            headers.append((key, res._headers[key]))
    
        # Create mock equivalent HTTP response
        _res = HTTPResponse(req.method, self._url(urlunparse(req.url)))
    
        # response status
        _res.version = (1, 1)
        _res.status = res._status
        _res.reason = http_reasons.get(res._status)
        _res._should_close = False
    
        # Add response headers
        _res.raw_headers = tuple(headers)
        _res.headers = multidict.CIMultiDictProxy(
            multidict.CIMultiDict(headers)
        )
    
        # Define `_content` attribute with an empty string to
        # force do not read from stream (which won't exists)
        _res._content = ''
        if res._body:
>           _res._content = res._body.encode('utf-8', errors='replace')
E           AttributeError: 'bytes' object has no attribute 'encode'

env\lib\site-packages\pook\interceptors\aiohttp.py:110: AttributeError

with string content no exception and all working as expected

[tomas-fp]

The binary body error is clearly a bug. I need to test the URL issue.

Will take a look later today.

[h2non]

I can't reproduce the URL bug. I'm testing both requests and aiohttp:

import pook
import requests
import aiohttp
import asyncio
import async_timeout


async def fetch(session, url):
    with async_timeout.timeout(10):
        async with session.get(url) as res:
            print('aiohttp response:', res.status)
            print('aiohttp body:', await res.text())


with pook.use(network=False):
    pook.get('http://data.alexa.com/data', times=2,
             reply=200, response_type='json',
             response_headers={'Server': 'nginx'},
             response_json={'foo': 'bar'})

    async def run():
        async with aiohttp.ClientSession() as session:
            await fetch(session, 'http://data.alexa.com/data?cli=10&dat=snbamz&url=http%3A%2F%2Ftest.org')

    # Test aiohttp
    loop = asyncio.get_event_loop()
    loop.run_until_complete(run())

    # Test requests
    res = requests.get('http://data.alexa.com/data?cli=10&dat=snbamz&url=http%3A%2F%2Ftest.org')
    print('requests response:', res.status_code)
    print('requests body:', res.text)

[h2non]

Body bytes issue should be fixed in pook@0.1.10. Feel free to upgrade it:

pip install -U pook

[sky-code]

Good job.
For url bug try this code

DATA_ALEXA_API_URL = 'http://data.alexa.com/data'
DATA_ALEXA_API_DEFAULT_PARAMS = {'cli': '10', 'dat': 'snbamz'}
async with self.http_session.get(DATA_ALEXA_API_URL, params=alexa_api_params) as resp:
    alexa_response_text = await resp.text()

Here pook will match url 'http://data.alexa.com/data' but not full url 'http://data.alexa.com/data?cli=10&dat=snbamz'

bug here in that fact i pass parameters but pook trying match url without parameters when using aiohttp but for requests same code will try to match full url with parameters

[tomas-fp]

I will debug that.

I think you can temporary workaround that defining the params at URL string level, like I shared above.

[sky-code]

At this moment i just match url without parameter as workaround, for me this is not critical bug.
Main problem most likely in params=alexa_api_params, looks like pook doesn't know about it

[h2non]

Fixed in pook@0.1.11. Just upgrade it:

pip install -U pook