Further `save-daily-metrics.yml` improvements
Opened this issue · 0 comments
e-belfer commented
Overview
What is the problem we're solving? For very simple items, this can be encapsulated in the success criteria.
The save_daily_metrics
run fails if the file already exists. It will also not run the Kaggle archiver if the Github archiver fails. Let's fix this!
Success Criteria
How will we know that we're done?
- If a file already exists, we catch the error rather than getting a "does not have storage.objects.delete access" error
- Switch to using a requests.get() method with retry logic in all saved archives
- Use a
matrix
to run each archiver simultaneously