Scrape all available data on Schedge v1
A1Liu opened this issue · 8 comments
Rescrape, to fix invalid/missing data from previous scrapes:
- fa2020
- ja2021
- sp2021
- su2021
- fa2021
- ja2022
- sp2022
- su2022
- fa2022
- ja2023
- sp2023
^ Need to re-do again because we're missing section titles and timezones are fucked
for time zones, need to store local time in DB and then dynamically convert when serving the data, because I don't see another way to not delete information.
^ Need to re-do again once the bug with course titles is fixed.
Should probably use a Github workflow to scrape; logging into lightsail is a bit silly
https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow
(done)
The only one that Schedge v1 has that Schedge v2 doesn't have is su2020
https://schedge.a1liu.com/2020/fa/ua/csci - available (available on v2)
https://schedge.a1liu.com/2020/su/ua/csci - available (missing from v2)
https://schedge.a1liu.com/2020/sp/ua/csci - missing
https://schedge.a1liu.com/2020/ja/ua/csci - missing
Need to add the --v1 flag to the scrape job, then deploy again
This is now done: