IIIF/website

robots.txt needs full URI for sitemap

hadro opened this issue · 1 comments

hadro commented

https://iiif.io/robots.txt

Our current robots.txt has a relative URL for the sitemap, but the Google Search Console is showing an error for our one-line robots.txt, and I believe that's because it needs a full path to the sitemap:
https://developers.google.com/search/docs/advanced/robots/create-robots-txt

It looks like the robots.txt should be added to the website repo as it was created by the sitemap Jekyll plug-in.

A question to think about. Should we add old versions of the spec so that they aren't indexed? Particularly the ones that appearing in #170