web-robots

There are 3 repositories under web-robots topic.

jonasjacek/robots.txt
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
87 7 038
din0s/ml-for-bot-detection
A Python notebook showcasing the use of Machine Learning for the task of bot detection, with an emphasis on e-commerce sites.
Language:Jupyter Notebook12 1 05
acuciureanu/spidertrap-rs
A simple trap for web crawlers
Language:Rust10 2 30