Instruct crawlers to avoid indexing tools pages
ahayes opened this issue · 0 comments
ahayes commented
Nunaliit dynamically serves robots.txt content pointing to sitemap.xml content. Somewhere in there, we should tell crawlers to avoid indexing /tools/*. Google seems to get grumpy about Soft 404 Errors when a page like the submission tool is accessed anonymously and shows some errors as a result.