glamod/glamod-nuim

monthly_to_cdm_all_v1.py issues

Closed this issue · 3 comments

sjnoone commented

I ran a test on the monthly cdm conversion and discovered the following isssues with the tables:
1)decimal places incorrect source_id column in the cdm_lite table should be 0 places not 2 (/ichec/work/glamod/data/level2/land/r6.0/cdm_lite/monthly)
2) decimal places incorrect source_id column in the cdm_observations table should be 0 places not 2 (/ichec/work/glamod/data/level2/land/r6.0/observations_tables/monthly)
3) Code is not writing any values to the header files. (/ichec/work/glamod/data/level2/land/r6.0/header_tables/monthly)
4) Output files are not .gz zipped

all the code and atskfarm fiel scan be found at:/ichec/work/glamod/data/level2/land/r6.0/code/monthly

rjhd2 commented

Noting issues

  1. source_id field didn't have the snippet to remove the ".0" at the end. As source_id created in add_data_policy, this is now handled here, line 190.
  2. As above
  3. data_policy_df was supplying poor source_id and record_id values (as floats, not ints), so the merging on columns wasn't working, and caused by a malformed record_id_mnth.csv. This is likely to have been the root cause of 1 & 2 as well. Have added lines to address this 269-70 in case it happens again.
  4. COMPRESSION was empty string, so now set to .gz
sjnoone commented

Thanks Robert, tested and runs great now. Note that the input .csv gsom files are not compressed so does this make a difference to the compression code?

rjhd2 commented

No, the changes are such that the input and output compression can be set separately. Something which might be worth doing for the daily and subdaily scripts too just for flexibility.

  • 👍1