mozilla/network-pulse

Staging Sites don't have no-index

Closed this issue · 3 comments

benhohner commented

Pulse staging doesn't have a no-index meta tag or robots.txt so it is showing up in google. This can be harmful as google may detect Pulse or Pulse staging as duplicate content and it may negatively impact our SEO.

See: https://moz.com/learn/seo/duplicate-content for more info

  • update build process to automatically include a no-index, whether it be through a meta tag, robots.txt or other.
  • verify it doesn't apply to the live site
  • check our other properties to make sure the same problem doesn't exist elsewhere and make tickets to fix if it does
jamilasnell commented

"Can we check if Heroku has this built in?!" - Pomax, 2021

Pomax commented

looks like this is not something that heroku does for us, but there is an X-Robots-Tag package for Django that lets us set noarchive/noindex on everything staging related. That said, it looks fairly new [1], and we might just want to slip in our own middleware instead if all we're doing is injecting a header on every possible view.

  1. https://pypi.org/project/django-x-robots-tag-middleware/
Pomax commented

I forgot that our Pulse server is not Django (only the Pulse API is), but uses Express.js, so the above-mentioned library is not applicable, but the solution there turns out to be a lot easier. PR filed.