elacuesta/scrapy-pyppeteer

error: _request_handler not being called

cruzdanilo opened this issue · 2 comments

_request_handler is not being called, hanging the execution until timeout.

i tried to fix it following pyppeteer documentation example and replaced this:

page.on("request", partial(_request_handler, scrapy_request=request, stats=self.stats))

with this:

page.on("request", lambda req: asyncio.create_task(
    _request_handler(req, scrapy_request=request, stats=self.stats)
))

that worked, but something is still off. i get some errors and timeout on page.goto.

WARNING [21-02-24 17:11:02] py.warnings _showwarnmsg :109 lib/python3.9/site-packages/pyee/_base.py:81: RuntimeWarning: coroutine '_response_handler' was never awaited
  f(*args, **kwargs)

am i getting something wrong here?

macOS v11.2.1
python v3.9.1
pyppeteer v0.2.5
scrapy v2.4.1
scrapy-pyppeteer v0.0.14

Seems odd, the last successful build uses pyppeteer==0.2.2. Evidently something changed in between, but I cannot figure exactly out what from the changelog.
I implemented your suggested fix, but for some reason it still times out in Python 3.6.

I released the above as 0.0.15, thanks for the report and the code.
However, please know that I will be stopping active development of this project in favour of scrapy-playwright, which has a very similar API.