Crawling a specific page

Question

Crawling a specific page

justaj opened this issue 9 years ago · 2 comments

Hey! I was wondering if there was a way to return all links found on a specific page. So far spidr has been great for crawling a whole site but with my testing I'd like to be able to focus on one page.

Thanks

Answer 1 · 2016-05-07T05:08:28.000Z

You could use every_page { |page| ... } and filter if page.url.path == '...' for the page you are looking for? Also, you could use visit_urls_like to only visit that specific page.

Answer 2 · 2016-05-07T05:14:39.000Z

Oh, you could also manually request the page via net-http, mechanize, RestClient, etc and manually create a Spidr::Page object.