/sitemap-sniffer

Crawler that will check for most common URLs where are sitemaps, which simplifies the crawling when you know the sitemap.

Primary LanguageJavaScript

Sitemap sniffer

Sitemap sniffer will check the most used variants of sitemaps and you can use that for crawling. This will just save you time so you don't have to check manually.

Run it on Apify