Can I crawl links in all pages
Closed this issue · 2 comments
tibotiber commented
Hi, this is a really cool tool, thanks for building it.
When I crawl the website, i'd like it to crawl all pages, not only the ones linked to from the first page. For example, if my index page links to a blog page and the blog page has 5 posts, i'd like for the 5 posts to be queued for reporting as well.
Is this possible at the moment?
Thanks!
TGiles commented
Thanks for the complement! To answer your question, the current crawler configuration is set up to crawl all links in a particular domain. The related documentation for simple crawler depth.
In other words, the tool will crawl all links, as long as the links are part of the initial domain(s).
Does that answer your question?