Feature for adding noindex, nofollow meta tags in non-production (OSOE-621)
Closed this issue · 4 comments
Staging and other non-production environments can be open to the public internet. To prevent search bots from indexing these (and thus duplicating production content), let's add a new module that when enabled, adds a noindex, nofollow meta tag when the app is not running in a production environment.
<meta name="robots" content="noindex, nofollow">The environment can be checked with IHostEnvironment.IsProduction(). However, add a separate configuration coming from a configuration provider that indicates if the current environment is production or non-production, in case ASP.NET Core should be told it runs in Production but not actually (like in staging, that should be as close to production as possible).
Any good name suggestions? SearchBotBlocker, NoIndexNoFollow, EnvironmentMetaTag. I think SearchBotBlocker is fairly accurate.
Must we use the meta tag? I mean this is easily solvable with a middleware, that adds a response header:
context.Response.Headers.Add("X-Robots-Tag", "noindex, nofollow");The difference is, that this way there will be no
<meta name="robots" content="noindex, nofollow">tag in the HTML file.
According to my research, there are no meaningful differences. The only difference is: "A robots header can also override a meta tag if they have conflicting values."
Actually I think it's better, since we can only use the meta tag for HTML files.
For the name, EnvironmentRobots?
I didn't know about the header, but I'd use both to be sure (adding the meta tag is just simply an IResourceManager call from a filter).
Okay, I agree. After some more research I discovered, that using both method is the best, since there are some cases where one is not supported.