Metacritic scraper to generate review datasets for games, movies, music, and tv
####Dependencies:####
- PHP CLI
- Bash
- Curl
This library is broken into two parts:
- A script to pull down and parse the Highest Rated pages from Metacritic and then generate an output file with the data
- A script to parse the results file and generate a pure CSV file of the fields then pull down and save the image associated with each title
xbox 360 reviews
"Grand Theft Auto IV","98","7.9","Apr 29, 2008","M","Rockstar Games",""
[Grand Theft Auto IV]http://img1.gamespotcdn.com/metacritic/public/www/images/products/games/3/78b2451531891db4396d873d82accbeb-98.jpg
"BioShock","96","8.7","Aug 21, 2007","M","2K Games",""
"The Orange Box","96","9.0","Oct 10, 2007","M","EA Games",""
[The Orange Box]http://img2.gamespotcdn.com/metacritic/public/www/images/products/games/5/a1e86e73c40d4ce70191b86ccbf9295f-98.jpg
"Mass Effect 2","96","9.0","Jan 26, 2010","M","Electronic Arts",""
[Mass Effect 2]http://img2.gamespotcdn.net/metacritic/public/www/images/products/games/2/7178937e2ea07ddd2ae3e46bdf746dc7-98.jpg
"Red Dead Redemption","95","8.9","May 18, 2010","M","Rockstar Games","Rockstar San Diego"
[Red Dead Redemption]http://img2.gamespotcdn.com/metacritic/public/www/images/products/games/1/1aba85e322bb75caf4fcc3f15528963e-98.jpg
"Gears of War","94","8.5","Nov 7, 2006","M","Microsoft Game Studios",""
[Gears of War]http://img1.gamespotcdn.net/metacritic/public/www/images/products/games/5/4e6253bf2efe800e05d4c2270a52a674-98.jpg
"The Elder Scrolls IV: Oblivion","94","8.8","Mar 20, 2006","M","2K Games",""
[The Elder Scrolls IV: Oblivion]http://img1.gamespotcdn.com/metacritic/public/www/images/products/games/2/180b7a5751235c7f2322eb652070c833-98.jpg
-
Pull first 10 pages of Xbox 360 reviews
./pull.sh 10 xbox360_reviews.txt -
Pull first page of PC reviews
./pull.sh 1 pc_reviews.txt games pc -
Pull first 5 pages of Movie reviews
./pull.sh 5 movies.txt movies
Required
1. Number of pages deep to parse off of the score browser (page 0 is the highest scores)
2. Name of the output file
Optional [case sensitive]
3. Media type
valid options: movies, games, music, tv
4. Sub-Media type (only applies when 'games' are specified)
valid options: ps3, xbox360, wii, pc, psp, ds, ps2, 3ds, ios