catalyst-cooperative/pudl-usage-metrics

Further `save-daily-metrics.yml` improvements

Opened this issue · 0 comments

Overview

What is the problem we're solving? For very simple items, this can be encapsulated in the success criteria.
The save_daily_metrics run fails if the file already exists. It will also not run the Kaggle archiver if the Github archiver fails. Let's fix this!

Success Criteria

How will we know that we're done?

  • If a file already exists, we catch the error rather than getting a "does not have storage.objects.delete access" error
  • Switch to using a requests.get() method with retry logic in all saved archives
  • Use a matrix to run each archiver simultaneously

Next steps