This will go around in circles crawling the same pages over and over.

Question

This will go around in circles crawling the same pages over and over.

superlowburn opened this issue 7 years ago · 3 comments

superlowburn commented 7 years ago

^^^

Answer 1 · 2017-07-06T20:32:44.000Z

Please provide an example or describe the problem further. The crawler should ignore already fetched pages.

Answer 2 · 2017-07-06T21:40:34.000Z

Hi,

./sitemap-generator -bq gbiz.org sitemap.xml --verbose

I repeatedly get the same URLs being crawled.
Too many to cutnpaste here.

Answer 3 · 2017-07-07T06:48:06.000Z

Started a crawl and looks like it does what it should do. It only adds pages a single time. Could you paste an example URL which is added more than once?

Also you don't need the -b flag if you are entry is the home page. It matches all pages anyways.