Support optional creation of separate, wordlist-like output for found links
cxosmo opened this issue · 4 comments
Thanks for putting out this great tool @xnl-h4ck3r.
I think it'd be useful to implement an optional feature which does a (small) amount of post-processing on found links, then writing this output to a separate, wordlist-like file in conventional format (e.g. line-separated example.html, api/v1/users etc). This would allow for easier chaining with other tools (e.g. passing this file to ffuf or Intruder) or creating/appending to custom wordlists using this output.
I'd be happy to submit a pull request which adds this feature if you think it'd be worthwhile (I implemented something similar on a private rework of GerbenJavado's LinkFinder and found it to be handy)!
Hi @cxosmo
What kind of post processing do you mean? Do you mean instead of the list oif URLs, you want a file with just paths, files, etc? I'm not sure I fully understand what you're after.
I shoudl probably make it so you can pipe xnLinKFinder so the output can go to another tool. You could then use unfrul or something to get paths, etc if that's what you mean?
Regards
Xnl
Since you mention unfurl, I'll use that to explain what I'm driving at. Using an output file (output.txt) from xnLinkFinder (containing filenames, absolute paths, relative paths, and URIs containing paths), this will output more or less what I'm suggesting re: post processing:
cat output.txt | unfurl format "%p%?%q" | sort -u
The above command should return a list of consistently-formatted* absolute paths which can then be passed onto other tools e.g. ffuf/Intruder cleanly.
* Consistently formatted in that every path now starts with a / and can be handled by a single payload marker (e.g. https://example.comFUZZ) - rather than having a mix of relative/absolute paths which would need to be "standardised" (for lack of a better term, e.g. either all made to start with/without / and having scheme://domain stripped) before being passed onto this process.
Supporting piping would be great as you say, since unfurl handles the functionality I'm suggesting above and some folks in the bug bounty space love one liners!
I've added in piping so that will be in the next release hopefully, so you'll be able to do python3 xnLinkFinder.py -i evil.com -d 2 | unfurl format "%p%?%q" | sort -u for example. It will still write the output file, but will pass through to piped processes too. Will that fully give you want you need?
Sounds good and will work for my use case -- thanks for your efforts on maintaining this great tool!