page endpoint
sushidelivery opened this issue · 2 comments
Hello! Thank you for your amazing contribution!
I would wanted to ask about using proxy with email & password?
How can I configure that?
Many thanks!
i got it. Just enter like this "http://user:pass@10.10.1.10:3128/"
Please, reply on the second question;)
and second question, is there anyway I can add a checkpoint, so if i scraped for first time, and then when i scrape for the second time it will start from page where first time ended?
No, there is no starting page option. You could implement this feature, and I would help, but it won't be easy. You would have to modify the ._next_page() method of every class that inherits from SearchEngine separately, because you would have to be able to ask for a specific pagination element and each search engine has different HTML structure.