alexander-irbis/robots_txt

Issues

• Got a panic with German umlauts

  #11 opened 5 years ago by tclaus
  2

• Upgrade URL to 2.0

  #14 opened 5 years ago by theduke
  0

• Follow the robots.txt specifications from Google.

  #13 opened 5 years ago by alexander-irbis
  0

• Yandex-style matcher with reorder directives by prefix len

  #4 opened 8 years ago by alexander-irbis
  0

• Ignore bad `Host` domain

  #3 opened 8 years ago by alexander-irbis
  0

• Support `Clean-param`

  #2 opened 8 years ago by alexander-irbis
  0

• Match against complex rules

  #1 opened 8 years ago by alexander-irbis
  0