Increase Script / Data reliability
franTarkenton opened this issue · 0 comments
franTarkenton commented
At the moment the script is setup to run a single date. Sometimes this doesn't work as the modis data or the viirs data is not available when the script runs.
This epic will implement the following:
- when the pipeline runs it will evaluate what data we already have and the data that is available and process based on that configuration
- Separate the steps taken in the dailypipeline so that if one step fails it doesn't impact other unrelated scripts.
- Reconfigure scripts so that they can automatically pickup where they last got to.
- when data is downloaded immediately push it back to object storage.
- before downloading data check to see if the data exists in ostore first
- general improvements to try to make the code more readable. (use utility methods with descriptive names vs file name / date / data hacking using string manipulation regex's etc.) Methods should also provide clear description of what they do with examples