elacuesta/scrapy-pyppeteer

Properly set pagecoroutines

scrapenetwork opened this issue · 2 comments

Hey im having trouble how to properly set

pagecoroutine and
Navigationpagecoroutine

using your example

def start_requests(self):
    yield scrapy.Request(
        url="http://quotes.toscrape.com/scroll",
        meta=dict(
            pyppeteer=True,
            pyppeteer_page_coroutines=[
                PageCoroutine("waitForSelector", "div.quote"),
                PageCoroutine("evaluate", "window.scrollBy(0, 2000)"),
                PageCoroutine("waitForSelector", "div.quote:nth-child(11)"),  # 10 per page
            ],
        ),
    )

does not pass start requests
Error
“Name” ‘pagecoroutine’ is not defined

Is it possible to edit for a working example?

Those examples don't include all the necessary imports for the sake of brevity. You should be able to solve the issue by adding:

from scrapy_pyppeteer.page import PageCoroutine, NavigationPageCoroutine

Updated in 4100ef9 and 037a5a5, closing.