cedadev/cf-checker

actual_range checking is not accurate for packed data

Opened this issue · 0 comments

if min_dv and actual_range[0] != min_dv:

I've run into an issue with packed data giving different actual_range values.

I calculate
pressure_msl:actual_range = 989.5993f, 1031.399f ;
in the code when creating the file, but then the checker gives me this error.

------------------
Checking variable: pressure_msl
------------------
ERROR: (2.5.1): First element of actual_range must equal minimum data value of variable after scale_factor/add_offset applied (989.56298828125)
ERROR: (2.5.1): Second element of actual_range must equal maximum data value of variable after scale_factor/add_offset applied (1031.401611328125)

As you can see, if you round these they are the same, but I think due to precision issues with packing and unpacking the data, the actual_range values don't match perfectly. So maybe this should be checked using an epsilon value or something like that. Here is an example where the unpacked values are very small numbers:

vorticity_low:actual_range = -2.90598e-05f, 2.69841e-05f ;
vs

------------------
Checking variable: vorticity_low
------------------
ERROR: (2.5.1): First element of actual_range must equal minimum data value of variable after scale_factor/add_offset applied (-2.9181901481933892e-05)
ERROR: (2.5.1): Second element of actual_range must equal maximum data value of variable after scale_factor/add_offset applied (2.7350401069270447e-05)