ekalinin/robots.js

Issues

• Access true while robots.txt disallows all

  #22 opened 9 years ago by basvdijk
  3

• HTTP request User-Agent header styled incorrectly

  #30 opened 7 years ago by midnightfreddie
  0

• praser.parse() throwing Cannot read property \'length\' of undefined'

  #29 opened 7 years ago by zackiles
  1

• Provide a CLI interface to test if a specific URL is accessible using a specific user agent

  #28 opened 7 years ago by gajus
  0

• Don't return first rule match for canFetch

  #18 opened 10 years ago by sebastianwessel
  4

• setUrl() is blocked

  #23 opened 9 years ago by fprivitera
  0

• 'Allow all' statement or blank line returns true for disallowed URLs

  #19 opened 10 years ago by sanderheilbron
  0

• Everything returns true...

  #17 opened 10 years ago by morganrallen
  3

• Ignoring rules without *

  #16 opened 10 years ago by SoAG
  5

• init parser from robot.txt string or allow to serialize the parser

  #12 opened 11 years ago by scherler
  1

• getCrawlDelay not public/documented?

  #11 opened 11 years ago by reezer
  2

• Nothing to see here.

  #10 opened 11 years ago by heath
  0

• Sitemap

  #7 opened 11 years ago by reezer
  5