web-robots

There are 4 repositories under web-robots topic.

  • jonasjacek/robots.txt

    Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

    877038

  • din0s/ml-for-bot-detection

    A Python notebook showcasing the use of Machine Learning for the task of bot detection, with an emphasis on e-commerce sites.

    Language:Jupyter Notebook13105

  • acuciureanu/spidertrap-rs

    A simple trap for web crawlers

    Language:Rust12230

  • jimsmart/progszy

    Progszy is a hard-caching HTTP(S) proxy server, for web robots.

    Language:Go00