robots.txt needs full URI for sitemap
hadro opened this issue · 1 comments
hadro commented
Our current robots.txt has a relative URL for the sitemap, but the Google Search Console is showing an error for our one-line robots.txt, and I believe that's because it needs a full path to the sitemap:
https://developers.google.com/search/docs/advanced/robots/create-robots-txt
glenrobson commented
It looks like the robots.txt should be added to the website repo as it was created by the sitemap Jekyll plug-in.
A question to think about. Should we add old versions of the spec so that they aren't indexed? Particularly the ones that appearing in #170