lgraubner/sitemap-generator-cli

Feature Request: "--retries" option

EpicLPer opened this issue · 0 comments

Heya,

Stumbled upon this script while trying to scrape a .onion site (with a bit of a workaround).
So far it's working okay, but due to the nature of Tor it sometimes happens that some sites report a 404 or other error codes.

For such cases a "--retries" option would be handy so the script tries by itself to index that specific URL again for x given attempts :)

Thanks!