artoo.ajaxSpider on dynamic data

Question

artoo.ajaxSpider on dynamic data

chrisvasey opened this issue 6 years ago · 9 comments

Hi there!

I have been using the artoo.waitFor helper to load dynamic content on a single page.
The issue I have faced is that when using the ajaxSpider method I am unable to get artoo to run with the .waitFor method as it just returns the HTML.

Is it possible for me to crawl the dynamic content or am I miss understanding?

Answer 1 · 2019-02-25T16:06:41.000Z

The content I am trying to scrape is set using JS so pulling the DOM does not help sadly

Answer 2 · 2019-02-25T16:09:19.000Z

Yes, dynamic content is the limit here. If what you retrieve is html you could try to inject it into an iframe or something but this is a longshot. Seems you need browser emulation at this point. Or a chrome/firefox extension taking control over your browser.

Answer 3 · 2019-02-25T16:12:43.000Z

Thank you, I suspected as much but wondered if anyone had ever achieved it within the library. Sadly the client I am using needs to fetch some data but I can’t install any dependencies only inject JS so other methods of scraping in Python/Node aren’t applicable. Thank you for your input!

Answer 4 · 2019-02-25T16:22:17.000Z

Sadly the client I am using needs to fetch some data

What is the format of this data? JSON? HTML? Have you tried retro-engineering the API?

Answer 5 · 2019-02-25T16:49:42.000Z

Sorry I could have been clearer - I need to fetch data from another page on the website using only injected JS snippets. Up till now this has worked fine with jQuery .load() and Ajax calls but now I am trying to do the same to fetch data that is dynamically rendered with JS so these tools do not return the loaded values. It may be possible to get the calls the page is making it’s self and attempt to pull the info from this but I would need to do this in a lot of places hence why scraping the exact values would have been ideal!

Answer 6 · 2019-07-22T22:31:46.000Z

Hi @chrisvasey,

Have you achieved to fetch dynamic data using artoo?

Answer 7 · 2019-07-23T06:48:20.000Z

Hi @santteegt, In the current setup I am not sure it is possible in JS.
I ended up using python to achieve this task which is a shame because I could not do it straight in the browser.

I will close this thread as I didn't realise it was still open.

Answer 8 · 2019-07-23T09:27:25.000Z

I was only able to use recursive with setInterval. If you use waitFor or loop, it doesn't work due to the fact that JS is an asynchronous language.

Answer 9 · 2019-07-23T09:28:43.000Z

If you use waitFor or loop, it doesn't work due to the fact that JS is an asynchronous language.

What do you mean? waitFor actually uses setInterval. A loop won't work indeed because it will freeze the main stack.