Small DC scraper problems
Opened this issue · 2 comments
stucka commented
The scraper assumes a link for 2014 is listed but echoes a different year. For 2024's index, at least, that is not the case; no link to 2014 is offered, and therefore the scraper will not automatically try to download 2014.
Scraper grabs old HTML even when there's no update, e.g., in 2024 maybe we want to update 2024 and 2023, but we don't need to grab 2019 again. A small optimization might be to try to hit older URLs with utils.fetch_if_not_cached and then read from it the files with cache.
stucka commented
I'm sure there was a reason to use this uuid thing instead of labeling the files with the years, but I'm not seeing it at the moment. =)
stucka commented
Also should maybe nix the http:// prefix in the use of 2014.