temoto/robotstxt

Issues

• TestAgent and Test (for the same user-agent) gives different results in case of temporary error when fetching the robots.txt file

  #40 opened 10 months ago by masonlouchart
  0

• List allow & disallow

  #38 opened 2 years ago by TheUltimateCookie
  2

• Add support for Crawl-Delay

  #22 opened 5 years ago by moredure
  5

• compatible for https://github.com/google/robotstxt

  #23 opened 2 years ago by exfly
  2

• Parser not getting sitemap from robots.txt

  #33 opened 3 years ago by TheUltimateCookie
  2

• sitemaps?

  #28 opened 4 years ago by hackermondev
  1

• Parse rules for a given user agent

  #24 opened 5 years ago by yields
  1

• panic: runtime error: invalid memory address or nil pointer dereference

  #20 opened 5 years ago by crosscue
  9

• Incorrect usage of $ symbols

  #21 opened 6 years ago by lmas
  6

• Change import statement or change package name

  #16 opened 8 years ago by sudotliu
  3

• Different behavior on Google Webmaster Tools robots.txt checker and robotstxt-go

  #15 opened 8 years ago by uforic
  3

• Crash with unicode symbols

  #12 opened 10 years ago by vad
  2

• No error is returned by FromResponse() when the file is incorrect

  #10 opened 12 years ago by alexander-bauer
  4

• Groups are not handled correctly

  #6 opened 12 years ago by mna
  13

• Reduce memory footprint of multiple logically same RobotsData acquired from different inputs

  #5 opened 12 years ago by temoto
  2

• Separate DNS resolve from connect

  #9 opened 12 years ago by temoto
  1

• fix '//' formatting in README

  #1 opened 14 years ago by temoto
  1

• parse non-empty robots.txt

  #2 opened 14 years ago by temoto
  1