OPENDAP/bes

Missing zlib (deflate) compression level from dmrpp files

Opened this issue · 6 comments

ayushnag commented

I am working on parsing dmrpp metadata files to match the format of zarr metadata files (parser PR here). The only piece of missing information I have encountered in dmrpp files is the zlib (deflate) compression level. It seems like the opendap dmrpp generator has this logic to retrieve and store this attribute as deflateLevel (see example below) but it is not implemented across many files. Many NASA datasets including SWOT L2 SSH, MUR L4 SST, and Ice-SAT are missing this field.

Example dmrpp file WITH zlib level

Example dmrpp file WITHOUT zlib level

ndp-opendap commented

You might check the version of the build_dmrpp application that was used to build the dmr++ files that lack the information you are seeking. This should be available in the metadata of the dmr++:

    <Attribute name="build_dmrpp_metadata" type="Container">
        <Attribute name="build_dmrpp" type="String">
            <Value>3.20.13-310</Value>
        </Attribute>
        <Attribute name="bes" type="String">
            <Value>3.20.13-310</Value>
        </Attribute>
        <Attribute name="libdap" type="String">
            <Value>libdap-3.20.11-93</Value>
     </Attribute>

It may be the case that these dmr++ files were created prior to the deflateLevel being included in the output. If so, we would like to know which NASA collections you encounter that lack this, as we may be able to get them to rebuild those files with the most recent build_dmrpp application.

ndp-opendap commented

After reviewing the examples I think we'll need to appeal to @kyang2014 for his insights on this.

ayushnag commented

I can point to some examples of NASA datasets/dmrpp that have this issue but I'm not sure how to find all of them. The file I linked earlier with deflateLevel was build_dmrpp version 3.20.13. Some of the files below are above that version number and still missing deflateLevel. Could it be a DAAC/collection level issue?

https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20240101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc.dmrpp

metadata:

        <Attribute name="build_dmrpp" type="String">
            <Value>3.20.13-664</Value>
        </Attribute>
        <Attribute name="bes" type="String">
            <Value>3.20.13-664</Value>
        </Attribute>
        <Attribute name="libdap" type="String">
            <Value>libdap-3.20.11-198</Value>
        </Attribute>

https://archive.swot.podaac.earthdata.nasa.gov/podaac-swot-ops-cumulus-protected/SWOT_L2_LR_SSH_2.0/SWOT_L2_LR_SSH_Expert_017_463_20240705T134333_20240705T143458_PIC0_01.nc.dmrpp

metadata:

        <Attribute name="build_dmrpp" type="String">
            <Value>3.21.0-272</Value>
        </Attribute>
        <Attribute name="bes" type="String">
            <Value>3.21.0-272</Value>
        </Attribute>
        <Attribute name="libdap" type="String">
            <Value>libdap-3.21.0-70</Value>
        </Attribute>

This file below does have deflateLevel

https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/VNP14A1.002/VNP14A1.A2024189.h00v08.002.2024190091116/VNP14A1.A2024189.h00v08.002.2024190091116.h5.dmrpp

metadata:

        <Attribute name="build_dmrpp" type="String">
            <Value>3.21.0-46</Value>
        </Attribute>
        <Attribute name="bes" type="String">
            <Value>3.21.0-46</Value>
        </Attribute>
        <Attribute name="libdap" type="String">
            <Value>libdap-3.21.0-27</Value>
        </Attribute>
jgallagher59701 commented
kyang2014 commented

I double checked the date I added the deflateLevel to the dmrpp file. It is Sep. 5th, 2023 with #809

And the tag for this pull request is https://github.com/OPENDAP/bes/releases/tag/3.20.13-845. which is bes-3.20.13-845.
So BES 3.20.13-664, the one listed as without having the deflateLevel, was generated before the time I added this feature. The other two examples that have deflateLevels were generated after the time I added this feature.

jgallagher59701 commented