Parameterize repectRobotsTxt
Closed this issue · 0 comments
emersonthis commented
Sometimes we use this tool on development sites where we need to disable respectRobotsTxt. We should add a flag to ignore robots.txt.
It would be great if we could do this in a way that lets us pass any simplecrawler config overrides inline.