error: _request_handler not being called
cruzdanilo opened this issue · 2 comments
_request_handler is not being called, hanging the execution until timeout.
i tried to fix it following pyppeteer
documentation example and replaced this:
page.on("request", partial(_request_handler, scrapy_request=request, stats=self.stats))
with this:
page.on("request", lambda req: asyncio.create_task(
_request_handler(req, scrapy_request=request, stats=self.stats)
))
that worked, but something is still off. i get some errors and timeout on page.goto
.
WARNING [21-02-24 17:11:02] py.warnings _showwarnmsg :109 lib/python3.9/site-packages/pyee/_base.py:81: RuntimeWarning: coroutine '_response_handler' was never awaited
f(*args, **kwargs)
am i getting something wrong here?
macOS v11.2.1
python v3.9.1
pyppeteer v0.2.5
scrapy v2.4.1
scrapy-pyppeteer v0.0.14
Seems odd, the last successful build uses pyppeteer==0.2.2
. Evidently something changed in between, but I cannot figure exactly out what from the changelog.
I implemented your suggested fix, but for some reason it still times out in Python 3.6.
I released the above as 0.0.15
, thanks for the report and the code.
However, please know that I will be stopping active development of this project in favour of scrapy-playwright
, which has a very similar API.