This is a script for scraping the Elite Prospects website. It will parse
each teams roster and stat page from each league you pass to it over the
range of years you provide as well. From there it will compile the data
into three | delimited files labeled goalie_stats, player_stats,
and roster. A fourth file errorfile.txt. will be create from the parsing
to log which files couldn't be succesfully parsed because there were
no stats/rosters to parse that season.
The script will also create three new directories output_file,
leaguepages, and teampages. Leaguepages will hold the html files
of each leagues main page that will be scraped to get the teams. It will also
store a teamids.json file that holds the league ids for every league on
elite prospects.
The output_file folder will be where your stat files and
error file will be written. And finally the teampages stores all the html
for each team per season. You can delete these once your stats are compiled
but if you need the info again you'll have to rescrape it.
Please do not abuse this as the people at Elite Prospects do work hard. This is for research purposes only.
The easiest way to run this program is to clone it directly from Github. If you have git change to the directory you want to download the repo to and type:
git clone https://github.com/mcbarlowe/prospectscrape.gitNext I would create a virtual environment and then install the neccesary packages
by using pip and the requirements.txt file with this command: pip install requirements.txt.
Once that's downloaded you'll need to change into the prospectscrape directory to then edit
the teamscrape.py script with whatever text editor you want.
There are two main variables in the main() function you'll need
to change depending on what leagues and years you want to scrape. The
first is the leagues variable. This is an empty list you will fill with
the leagues you want to scrape. So if you want to just scrape say the
KHL you would change it to ['KHL']. If you wanted to scrape Liiga
and the QMJHL, you would need to change it to ['Liiga', 'QMJHL'] and
so on.
The next variable you will need to change the year variables in this
function: scrape_team_page(url_base, leagues, 2003, 2018). The first year
is the start year you want to begin scraping. Once those two isues are
set to the parameters you want then save the file and run the script like
you normally would. Happy scraping.
One more word of advice this is a slow process especially as you try to scrape multiple leagues/seasons in one go. I've made it slower by making the program wait randomly between scrapes so just don't expect it to be quick.