[REQUEST] Add option to parse only robots.txt files

Question

Opened this issue 4 years ago · 4 comments

Really quick tool, it would be great if it parses only robots.txt file

Answer 1 · 2021-06-06T21:20:24.000Z

@mr0xE I intended for the tool to be passive

Answer 2 · 2021-06-15T11:29:43.000Z

Add option to blacklist certain file types e.g png,ico,jpg,css

Answer 3 · 2021-06-16T07:34:20.000Z

Great suggestion, I will work on it

Answer 4 · 2021-06-20T19:30:55.000Z

Add option to blacklist certain file types e.g png,ico,jpg,css

@mr0xE , I added a -f, --filter option