RMP Review Scraping
Closed this issue · 4 comments
For now, I got some code for scraping the reviews. Though, for some professors, there are a lot of them. So you would have to simulate the button click on Load more and scrape all the review. This should be fairly straightforward. I can pass onto anyone who's interested the code I have so far.
You can also try making the request to this URL instead to avoid complications, eg: https://www.ratemyprofessors.com/paginate/professors/ratings?tid=1134872&filter=&courseCode=&page=1
It looks like each page serves 20 ratings, and the URL you provided actually provides the number of remaining ratings as part of the data. This should make it a lot easier to scrape through it programmatically.
Yep. I did it the more painful way before but figure that RMP offers an API endpoints so might as well. Feel free to implement it
This can be achieved in the frontend using the API above. Don't think this is necessary for us to scrape and store these info