CFchecker does not find a coordinate variable
Opened this issue · 20 comments
I am quite newcomer to CFChecker, so please forgive me for possible irrelevance of this issue
When provided with a file showing this structure (here abridged), CFchecker produces the output reported further below, which seems to be a misbehaviour
netcdf ch4global_Amon_CNRM-CM6-1_amip_r99i1p1f2_gr_197901-217812 {
dimensions:
axis_nbounds = 2 ;
str_len = 255 ;
time = UNLIMITED ; // (1 currently)
variables:
char sector(str_len) ;
sector:name = "sector" ;
sector:standard_name = "region" ;
float ch4global(time) ;
....
ch4global:coordinates = "sector" ;
....
Output :
Using CF Checker Version 3.0.5
Checking against CF Version CF-1.6
Using Standard Name Table Version 49 (2018-02-13T08:44:33Z)
Using Area Type Table Version 6 (22 February 2017)
ERROR: (2.6.1): This netCDF file does not appear to contain CF Convention data.
ERROR: (5): coordinates attribute referencing non-existent variable
this is also true when the main variable has more (space) dimensions
Hmmm. I've tried your snippet above and it works ok for me. Could you send me the header of your netcdf file please (ncdump -h) as I think there must be something elsewhere in the file that is causing this error. I will also improve that error message so that it indicates which variable is the problem.
Here it is. The file is small, I send it separately.
netcdf ch4global_Amon_CNRM-CM6-1_amip_r99i1p1f2_gr_197901-217812 {
dimensions:
axis_nbounds = 2 ;
str_len = 255 ;
time = UNLIMITED ; // (1 currently)
variables:
char sector(str_len) ;
sector:name = "sector" ;
sector:standard_name = "region" ;
double time(time) ;
time:axis = "T" ;
time:standard_name = "time" ;
time:long_name = "Time axis" ;
time:calendar = "gregorian" ;
time:units = "days since 1850-01-01 00:00:00" ;
time:time_origin = "1850-01-01 00:00:00" ;
time:bounds = "time_bounds" ;
double time_bounds(time, axis_nbounds) ;
float ch4global(time) ;
ch4global:online_operation = "average" ;
ch4global:cell_methods = "area: time: mean" ;
ch4global:interval_operation = "900 s" ;
ch4global:interval_write = "1 month" ;
ch4global:_FillValue = 1.e+20f ;
ch4global:missing_value = 1.e+20f ;
ch4global:coordinates = "sector" ;
ch4global:standard_name = "mole_fraction_of_methane_in_air" ;
ch4global:description = "Global Mean Mole Fraction of CH4" ;
ch4global:long_name = "Global Mean Mole Fraction of CH4" ;
ch4global:history = "none" ;
ch4global:units = "1e-09" ;
// global attributes:
:name = "/scratch/utmp/ftdir/senesi/eclis/transfers/toy_atm/iox/ch4global_Amon_CNRM-CM6-1_amip_r99i1p1f2_gr_%start_date%-%end_date%" ;
:Conventions = "CF-1.7 CMIP-6.2" ;
:creation_date = "2018-02-28T08:45:37Z" ;
:tracking_id = "hdl:21.14100/835f2c5e-469d-4fd6-9e3c-4edd6ff4fc20" ;
:description = "DECK: AMIP" ;
:title = "CNRM-CM6-1 model output prepared for CMIP6 / CMIP amip" ;
:activity_id = "CMIP" ;
:contact = "contact.cmip@meteo.fr" ;
:data_specs_version = "01.00.21" ;
:dr2xml_version = "0.99" ;
:CMIP6_CV_experiments_latest_tag = "3.3.0 (22; g8906e28)" ;
:experiment_id = "amip" ;
:experiment = "AMIP" ;
:external_variables = "areacella" ;
:forcing_index = 2 ;
:frequency = "mon" ;
:further_info_url = "https://furtherinfo.es-doc.org/CMIP6.CNRM-CERFACS.CNRM-CM6-1.amip.none.r99i1p1f2" ;
:grid = "data regridded to a T127 gaussian grid (128x256 latlon) from a native atmosphere T127l reduced gaussian grid" ;
:grid_label = "gr" ;
:nominal_resolution = "250 km" ;
:history = "none" ;
:initialization_index = 1 ;
:institution_id = "CNRM-CERFACS" ;
:institution = "CNRM (Centre National de Recherches Meteorologiques, Toulouse 31057, France), CERFACS (Centre Europeen de Recherche et de Formation Avancee en Calcul Scientifique, Toulouse 31057, France)" ;
:license = "CMIP6 model data produced by CNRM-CERFACS is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at http://www.umr-cnrm.fr/cmip6/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;
:mip_era = "CMIP6" ;
:physics_index = 1 ;
:product = "model-output" ;
:realization_index = 99 ;
:realm = "atmos atmosChem" ;
:references = "A character string containing a list of published or web-based references that describe the data or the methods used to produce it.Typically, the user should provide references describing the modelformulation here" ;
:source = "CNRM-CM6-1 (2017): aerosol: prescribed monthly fields computed by TACTIC_v2 scheme atmos: Arpege 6.3 (T127; Gaussian Reduced with 24572 grid points in total distributed over 128 latitude circles (with 256 grid points per latitude circle between 30degN and 30degS reducing to 20 grid points per latitude circle at 88.9degN and 88.9degS); 91 levels; top level 78.4 km) atmosChem: OZL_v2 land: Surfex 8.0c ocean: Nemo 3.6 (eORCA1, tripolar primarily 1deg; 362 x 294 longitude/latitude; 75 levels; top grid cell 0-1 m) seaIce: Gelato 6.1" ;
:source_id = "CNRM-CM6-1" ;
:source_type = "AGCM" ;
:sub_experiment_id = "none" ;
:sub_experiment = "none" ;
:table_id = "Amon" ;
:variable_id = "ch4global" ;
:variant_info = ". Information provided by this attribute may in some cases be flawed. Users can find more comprehensive and up-to-date documentation via the further_info_url global attribute." ;
:variant_label = "r99i1p1f2" ;
:EXPID = "toy_atm" ;
:CMIP6_CV_version = "cv=3.2.7-89-gac4b169" ;
:xios_commit = "1442" ;
:nemo_gelato_commit = "417012b853ea50e_c28a02af91d965f" ;
:arpege_minor_version = "6.3.1" ;
}
The file you sent me works absolutely fine through the head of master. I will leave this issue open for the moment so that once I've got a release candidate available for community testing you can confirm that it is ok for you.
Whilst testing this I did find one non-related bug - where the checker fell over due to not liking the one dimensional character array for sector. This has been fixed.
Thanks
@RosalynHatcher we're also getting ERROR (2.6.1): This netCDF file does not appear to contain CF Convention data. is there an obvious fix for this?
@durack1 @RobertPincus @taylor13
Without seeing the file I can't tell you what the problem is or even if it is related to this ticket. Could you either send me or make available the troublesome file please and I'll take a look.
@RosalynHatcher : I've just come across this. The problem may be that @senesis is using a version of the checker which defaults to CF-1.6 and doesn't support CF-1.7. In any case, the output above says that it is using CF-1.6, and in CF-1.6 the Conventions setting CF-1.7 CMIP-6.2 is invalid.
Here's an ncdump of the file:
netcdf rld {
dimensions:
expt = 18 ;
site = 100 ;
level = 61 ;
variables:
float lat(site) ;
lat:long_name = "ERA-Interim latitude" ;
lat:units = "degree_north" ;
lat:standard_name = "latitude" ;
float lon(site) ;
lon:long_name = "ERA-Interim longitude" ;
lon:units = "degree_east" ;
lon:standard_name = "longitude" ;
float time(site) ;
time:long_name = "ERA-Interim fractional day of the year 2014" ;
time:units = "days since 2014-1-1 0:0:0" ;
time:standard_name = "time" ;
time:calendar = "gregorian" ;
float plev(site, level) ;
plev:standard_name = "air_pressure" ;
plev:units = "Pa" ;
plev:coordinates = "lon lat" ;
plev:long_name = "Pressure at layer edge" ;
float rld(expt, site, level) ;
rld:variable_id = "rld" ;
rld:standard_name = "downwelling_longwave_flux_in_air" ;
rld:units = "W m-2" ;
rld:coordinates = "lon lat time" ;
float profile_weight(site) ;
profile_weight:title = "profile weight to recover global mean" ;
profile_weight:units = "1" ;
profile_weight:coordinates = "lon lat" ;
profile_weight:long_name = "profile weight to recover global mean" ;
// global attributes:
:activity_id = "RFMIP" ;
:product = "model-output" ;
:experiment_id = "rad-irf" ;
:table_id = "Efx" ;
:frequency = "fx" ;
:sub_experiment_id = "none" ;
:data_specs_version = "01.00.12" ;
:physics_index = 1LL ;
:Conventions = "CF-1.7 CMIP-6.0" ;
:mip_era = "CMIP6" ;
:experiment = "rad_irf" ;
:sub_experiment = "none" ;
:realization_index = 1LL ;
:initialization_index = 1LL ;
:source_type = "RAD" ;
:nominal_resolution = "10 km" ;
:realm = "atmos" ;
:grid_label = "gn" ;
:grid = "columns sampled from ERA-Interim, radiative fluxes computed independently" ;
:institution_id = "AER" ;
:institution = "Research and Climate Group, Atmospheric and Environmental Research, 131 Hartwell Avenue, Lexington, MA 02421, USA" ;
:source_id = "LBLRTM-12-8" ;
:version = "12.8" ;
:source = "LBLRTM 12.8 (2017): \naerosol: none\natmos: none\natmosChem: none\nland: none\nlandIce: none\nocean: none\nocnBgchem: none\nseaIce: none" ;
:further_info_url = "https://furtherinfo.es-doc.org/CMIP6.AER.LBLRTM-12-8.rad-irf.none.r1i1p1f1" ;
:forcing_index = 1LL ;
:license = "CMIP6 model data produced by AER is licensed under a Creative Commons Attribution "Share Alike 4.0" International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file). The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;
:creation_date = "2019-02-04T23:18:42Z" ;
:tracking_id = "hdl:21.14100/3f590ab5-4d3f-4a42-b2a2-0edc7a3d35b8" ;
:variant_label = "r1i1p1f1" ;
:variable_id = "rld" ;
}
@RosalynHatcher I'd note that for the global_attribute :Conventions = "CF-1.7 CMIP-6.0" ; this can have a valid sub-index of CMIP-6.0, CMIP-6.1 or the current revision CMIP-6.2 following the specs doc (findable at https://goo.gl/v1drZl, and current version which is 6.2.7)
I hadn't seen integer values expressed as 1LL. Is this o.k.? what does "LL" mean in this context?
Once I'd removed the LL's Karl mentions above in order to generate the netcdf. (I've not come across this either.) The file is read absolutely fine by cf-checker-3.1.0. The CF Checker doesn't care about the sub-index of anything in the Conventions global attribute other than CF-. Can you please confirm that you were using the latest version of the checker?
@RosalynHatcher I am using the web interface at http://puma.nerc.ac.uk/cgi-bin/cf-checker.pl. I've just confirmed that the checker doesn't seem to be parsing the Conventions attribute correctly:
WARNING: Cannot determine CF version from the Conventions attribute; checking against latest CF version: CF-1.6
Using CF Checker Version 2.0.9
Checking against CF Version CF-1.6
Using Standard Name Table Version 63 (2019-02-05T13:57:21Z)
Using Area Type Table Version 9 (07 August 2018)
@taylor13 Might LL refer to long ints? These are set as part of a Python dictionary and written using netcdf4:
expt_attrs = {
"Conventions" :"CF-1.7 CMIP-6.0",
"realization_index" :1,
"initialization_index":1,...
certainly possible. But @RosalynHatcher thinks that's causing problems with the checker. Rosalyn, does the checker accept long ints?
The version you are using is an old version that only goes up to CF-1.6. As Martin said "CF-1.6 CMIP-6.0" was invalid at CF-1.6, hence the failure. Your file contains CF-1.7 data so you should be using the latest checker on our test server: http://pumatest.nerc.ac.uk/cgi-bin/cf-checker.pl
Please try running it through this version. I don't know whether it will handle the LL though, I suspect not. The reason I had to remove them from the CDL you posted was because ncgen was giving me syntax errors because of it and I thus couldn't generate the netCDF file with LL in it.
The current puma.nerc.ac.uk server will be retired in the next couple of months sorry for the confusion, it's taken us much longer than anticipated to move across.
If you still have problems running it through the CF-1.7 version of the checker, can you please create a new issue as this isn't related to the original poster's one. Thanks.
Not sure if edits of messages get communicated out, but I just updated my mistake in the above that it isn't a bug rather that "CF-1.7 CMIP-6.0" is not valid at CF-1.6.
If the "LL" is still included next to integers, I would try to remove them. It's not good if ncgen doesn't know how to handle them. That would lead me to believe that they are not standard netCDF. (Even if the CF-checker doesn't object.)
@RobertPincus what software are you using to write these files? netcdf4-python no?