markerikson/react-redux-links

Obscure crawler matching

hrbonz opened this issue · 1 comments

hrbonz commented

A few things that could make the scripts harder to detect:

  • implement a simili-random scheduling interval to avoid strict patterns in scraping
  • random/multiple User-Agents
hrbonz commented

Sorry, wrong repo ::/