💡 Feature request: ROBOTS_DISABLE_INDEXING should use noindex
Closed this issue · 0 comments
kotx commented
Is your feature request related to a problem? Please describe.
Certain search engines don't respect robots.txt and only follow noindex.
Describe the feature you would like to be implemented
Inject
<meta name="robots" content="noindex, nofollow">
into the head of every page served.
Describe alternatives you've considered
Manually implementing user-agent/IP blacklisting on a reverse proxy.
Additional context / screenshot
Example: https://safe.search.brave.com/help/brave-search-crawler