`User-agent: * Disallow: /` makes yacy not index the page even if it's own user agent is allowed in `robots.txt`
GamePlayer-8 opened this issue · 0 comments
GamePlayer-8 commented
Good afternoon again,
There's an issue with YaCy robots.txt detection where if I put:
User-agent: yacybot
Allow: /
User-agent: *
Disallow: /
it will take the second definition as more important even if it's own user agent is allowed.