Using a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; uro aims to solve that.
It doesn't make any http requests to the URLs and removes:
- human written content e.g. blog posts
- urls with same path but parameter difference
- incremental urls e.g.
/cat/1/and/cat/2/ - image, js, css and other static files
First, install uro with pip:
pip3 install uro
Now, there's just one way to use it, no args, no bullshit.
cat urls.txt | uro
