whatwg/html-build

robots.txt not deployed

Closed this issue · 3 comments

annevk commented

I tried to figure out why this did not happen, but failed. My suggestion would be to modify the build script to generate one similar to https://url.spec.whatwg.org/robots.txt but without the branch-snapshots/ line as HTML doesn't have that. After that we can remove the file from whatwg/html.

foolip commented

It looks like there isn't anything that copies robots.txt into html-build/output/

annevk commented

I see, so this could be fixed somewhere in https://github.com/whatwg/html-build/blob/master/build.sh#L455?

Do we want to fix this by writing out a robots.txt resource there or do we want to keep a static one in whatwg/html? My preference is to write out one in the build script as this matches what we do for other standards.

foolip commented

My preference is to write out one in the build script as this matches what we do for other standards.

Me too.