A tool which scrapes public github repositories for common naming conventions in variables, folders and files
This current upload contains the results from 5,256,950 php files from 39,069 different repositories
Gitscraper examines PHP files to create SecList / Dictionary files which can be used against any envinonment ( not just PHP ) for pentesting & bounty hunters.
php gitscraper.php {GitHub Username} {GitHub Personal KEY}
It will collect the following:
-
Folder & File Names
-
GET & POST variables
-
HTTP Header Variables
-
Laravel GET,POST,PUT & DELETE Routes
Each time one of the above is found it is added into its appropriate file in the /raw directory. These raw files are then sorted and ordered by the most duplicated content at the top and then cleaned as best as possible and put in the /cleaned directory
Current command for cleaning file
sort raw/{filename}.txt | uniq -c -d | sort -n -r | sed '/^[[:alnum:]/-._ ]*$/!d' | cut -c 9- | sed '/^$/d' > cleaned/{filename}.txt
- ( sort ) Sort the file so all occurences are next to each other
- ( uniq ) Pick out all the uniqe lines and prepend with the number of times found
- ( sort ) Sort by the instance of times found to the most common is at the top
- ( sed ) Remove any lines that don't contain alphanumeric text, hyphens underscores, full stops and spaces.
- ( cut ) Remove the instances found number
- ( sed ) Remove any blank lines
- ( > ) Echo out to cleaned directory