Information gathering & website reconnaissance
Usage:
./extractor.sh http://www.target.org/
- IP and hosting info like city and country (using FreegeoIP)
- DNS servers (using dig)
- ASN, Network range, ISP name (using RISwhois)
- Load balancer test
- Whois for abuse mail (using Spamcop)
- PAC (Proxy Auto Configuration) file
- Compares hashes to diff code
- robots.txt (recursively looking for hidden stuff)
- Source code (looking for passwords and users)
- External links (frames from other websites)
- Directory FUZZ (like Dirbuster and Wfuzz - using Dirbuster) directory list)
- URLvoid API - checks Google page rank, Alexa rank and possible blacklists
- Provides useful links at other websites to correlate with IP/ASN
- Option to open ALL results in browser at the end
- Abuse mail using lynx istead of
curl - Target server name parsing fixed
- More verbose about HTTP codes and directory discovery
- MD5 collection for IP fixed
- Links found now show unique URLs from array
- [New feature] Google results
- [New feature] Bing IP check for other hosts/vhosts
- [New feature] Opened ports from Shodan
- [New feature] VirusTotal information about IP
- [New feature] Alexa Rank information about $TARGET_HOST
Tested on Kali light mini AND OSX 10.11.3 with brew
sudo apt-get install bc curl dnsutils libxml2-utils whois md5sha1sum lynx -y
Configuration file:
CURL_TIMEOUT=15 #timeout in --connect-timeout
CURL_UA=Mozilla #user-agent (keep it simple)
INTERNAL=NO #YES OR NO (show internal network info)
URLVOID_KEY=your_API_key #using API from http://www.urlvoid.com/
FUZZ_LIMIT=10 #how many lines it will read from fuzz file
OPEN_TARGET_URLS=NO #open found URLs at the end of script
OPEN_EXTERNAL_LINKS=NO #open external links (frames) at the end of script
- Upload to github :)
- Integration with other APIs
- Add host regex validation
- Use GNU parallel to fuzz URLs
- Export to CSV
- Possible migration to python
- Integration with JoomScan/WPScan/CMSmap
- Integration with CipherScan
- Check for installed packages
Download: Click Here (URLExtractor-master.zip)
Clone: git clone https://github.com/The404Hacking/URLExtractor.git