Using robots.txt to ignore subfolders?
Closed this issue · 1 comments
MapZombie commented
I have certain subfolders that need to be ignored such as "confidential" and "Archive". Can this be done with the robots.txt file? I have tried a number of combinations that would normally work with a website to no avail. Please advise on the purpose of robots.txt with harvester and on how I could implement this. (Tried with WAF and UNC)
User-agent: *
Disallow: /Archive/
pandzel-zz commented
robots.txt is a way to go, but only for WAF (UNC doesn't have a concept of robots). File robots.txt has to be accessible from the root of the server like http://myserver:myport/robots.txt