Update Scraper Courses Source
Closed this issue · 5 comments
Describe the solution you'd like
Previously, we relied on UofT Course Finder as our primary source for scraping courses. However, UofT is no longer updating the UTSG courses on this website (or at least very delayed updates).
UofT Course Finder also had other issues such as displaying incorrect UTM sections (eg. Sections that's closed until further notice). It would be ideal to move away from this website.
Describe Possible Solutions
- For UTSG courses, we could use this website instead. It provides many benefits such as a publicly accessible API, saving us the trouble of writing a scrapper.
- For UTM courses, we unfortunately need to rewrite our current scraper. I wrote a scraper before for the UTM Timetable Planner, but it has some drawbacks such as slow scraping time. We can improve on the scraping time significantly by using the puppeteer-cluster library (similar to the scraper for UofT Course Finder).
- UTSC also have their own timetable planner, but I'm not sure what it's called.
For UTSG, maybe we could leverage this: https://timetable.iit.artsci.utoronto.ca/api/20219/courses?code=eco
I can get someone on this but it'd have to be starting in August - so if anyone from the OG viaplanner team can tackle this before then that'd be great!
I can get someone on this but it'd have to be starting in August - so if anyone from the OG viaplanner team can tackle this before then that'd be great!
Yep sounds good, but it will probably take me some time 😅
Update: The UTSG Course scraper has been complete.