bopoda/robots-txt-parser

Issues

• Wrong boolean value in RobotsTxtValidator.php?

  #63 opened a year ago by scott8035
  1

• Remove asterisk (*) at the end of path.

  #15 opened 8 years ago by LeMoussel
  3

• Check valid content

  #24 opened 8 years ago by LeMoussel
  2

• Order of precedence for google user agents

  #46 opened 3 years ago by kennylajara
  7

• Crawl Delay of 0 will get ignored

  #59 opened 3 years ago by offsky
  1

• Fix deprecation about usort

  #60 opened 3 years ago by lyrixx
  0

• crowndiamond.org/robots.txt why empty rules parsed?

  #40 opened 6 years ago by bopoda
  1

• Could you publish this package to packagist ?

  #53 opened 6 years ago by lyrixx
  6

• Problems with iconv()

  #43 opened 8 years ago by LeMoussel
  2

• Sorted robots.txt

  #13 opened 8 years ago by LeMoussel
  2

• Check valid path

  #9 opened 8 years ago by LeMoussel
  6

• Directive with '+' not handled correctly

  #48 opened 8 years ago by LeMoussel
  1

• Typo on directive

  #16 opened 8 years ago by LeMoussel
  6

• Rules not handled correctly

  #27 opened 8 years ago by LeMoussel
  6

• Language and version

  #42 opened 8 years ago by kennylajara
  3

• Poor perfomance when parsing large file.

  #20 opened 8 years ago by LeMoussel
  2

• Some robots.txt files for robustness testing

  #17 opened 8 years ago by LeMoussel
  1

• Remove comments in content

  #25 opened 8 years ago by LeMoussel
  1

• isUrlAllow() not handled correctly if no path in Url.

  #30 opened 8 years ago by LeMoussel
  1

• Rules not handled correctly if dont't find User Agent

  #34 opened 8 years ago by LeMoussel
  0

• Rule: host not handled

  #32 opened 8 years ago by LeMoussel
  1

• Rule: noindex not handled

  #26 opened 8 years ago by LeMoussel
  1

• need to have 2 methods (isUrlAllow, isUrlDisallow) to check is url allow or not

  #10 opened 8 years ago by bopoda
  1

• IsUrlAllow Google test

  #14 opened 8 years ago by LeMoussel
  8

• Poor performance isUrlAllow() function

  #12 opened 8 years ago by LeMoussel
  6

• Не обрабатывать громадные файлы robots.txt

  #2 opened 8 years ago by bopoda
  1

• Get Url sitemaps

  #7 opened 8 years ago by LeMoussel
  3

• Suppress space in User_Agent

  #5 opened 8 years ago by LeMoussel
  1

• Правильно обрабатывать абсолютные urls в директивах allow/disallow

  #1 opened 10 years ago by bopoda
  1