Can i scrap javascript based website by this package?
touhi13 opened this issue · 1 comments
touhi13 commented
Can i scrap javascript based website by this package?
aleksa-sukovic commented
You need a custom user agent. Check this documentation section.
If you are scraping single page application (SPA), by default, it's not possible to "wait" for the JavaScript to generate the entire page.
However, by "pretending" to be one of the known web crawlers (such as Google or Yahoo) you might be able to fetch pre-rendered HTML. Note that this depends entirely on the website you want to scrape - some offer pre-rendered HTML to crawlers, some don't.