Respect Robots.txt Files

[mgifford]

Sites should respect the robots.txt files that some sites use to manage traffic.

Would be great if by default the scanner respected the wishes of the site owner.

[younglim]

We have developed a feature to follow robots.txt with -r flag when running node cli

  -r, --followRobots                 Option for crawler to adhere to robots.txt
                                     rules if it exists
                                 [string] [choices: "yes", "no"] [default: "no"]