Daily updates - land data

[agstephens]

@jhaigh0 As discussed with Simon, here is our plan for running daily updates:

1. Outline:

• we want a single script that will:
  • find the latest cdmlite daily update files
  • attempt to process each file:
    1. Restructure file to PSV under: /work/scratch...
    2. Generate SQL command to load the new file
    3. Execute the SQL command
    4. Remove the input file(s) under: /work/scratch...

2. Data locations and proposed workflow:

• data will be found in:
  • /gws/nopw/j04/c3s311a_lot2/data/level2/land/r202102/daily_updates/cdm_lite/
  • In the subdirectory incoming/
• for each file we process, we will:
  1. Move it to processing/
  2. Attempt to process it (as outlined above)
  3. If any part of the process fails, move it failed/ and log/email result
  4. If it succeeds, move it to complete/ and log/email result

3. Script details:

• the script can be bash or python
• it should be called something like: ./scripts/land/run-daily-update-cdmlite.(py|sh)
• it will need to use the settings file to get the location of the .../daily_updates/cdm_lite/ directory for a given release.

4. Once it is all working:

• we will schedule it to run once per day - probably submitted to a LOTUS node, but kicked off by the cron server.