Integrate All Steps with Object Storage

Question

Integrate All Steps with Object Storage

Opened this issue 2 years ago · 1 comments

When the script is run want it to be able to pickup from previous failures.

With the download process after data is downloaded it is copied to object storage, immediately after
When the download process starts before it attempts to retrieve data from object storage first before going to its official sources
When processing starts, it determines what needs to be done based on what data exists in the object storage

Answer 1 · 2023-05-25T20:53:56.000Z

started implementing this with the modis pipeline. Requires knowledge of the input /output files from each process. Due to the way the code is constructed this is going to be a bigger job then was initially anticipated.