TGiles/auto-lighthouse

Can I crawl links in all pages

Closed this issue · 2 comments

Hi, this is a really cool tool, thanks for building it.

When I crawl the website, i'd like it to crawl all pages, not only the ones linked to from the first page. For example, if my index page links to a blog page and the blog page has 5 posts, i'd like for the 5 posts to be queued for reporting as well.

Is this possible at the moment?

Thanks!

Thanks for the complement! To answer your question, the current crawler configuration is set up to crawl all links in a particular domain. The related documentation for simple crawler depth.

In other words, the tool will crawl all links, as long as the links are part of the initial domain(s).

Does that answer your question?

Thanks for the answer @TGiles, yes that's exactly the behaviour I expected. I don't know why it behaved like maxDepth was equal to 1 the first time I tried, but i've tested again this morning and I see that it's working as you described now. 🤷‍♂️