'Phony_dim' in .nc4 downloads of 3IMERGM data from Hyrax 1.13.3

Question

'Phony_dim' in .nc4 downloads of 3IMERGM data from Hyrax 1.13.3

ffang10 opened this issue 8 years ago · 9 comments

Although the (HDF5) handler in Hyrax 1.13.3 correctly presents the dimensions, a .nc4 download changes them to 'phony_dim_' in download file - this is in fact observed in other data like Merra2 as well. The 3IMERGM data is located here:

https://gpm1.gesdisc.eosdis.nasa.gov/data/s4pa/GPM_L3/GPM_3IMERGM.03/

The other issue with 3IMERG is that an 'nc_' is prepended to the download filename, a very undesirable feature to us. I recognize that both issues could be related to the netcdf library.

Answer 1 · 2017-02-28T15:23:19.000Z

I don't know much about the phony_dim in the fileout_netcdf response. I'll have to look at that.

I do know that the "nc_" prefix has something to do with the netCDF library not supporting file names that begin with a number. This limitation may be out of date, and if so the sever could change this behavior. Again, I'll have to see.

Answer 2 · 2017-02-28T15:35:45.000Z

Thanks Nathan. Yes prepending ‘nc_’ is likely a netcdf library behavior. In some of the (earlier) releases this behavior wasn’t there, or at least got suppressed. We built our secondary services on top of it. The non-uniform name change would affect those services to such a level that we probably can never upgrade Hyrax servers where such data is served. I wonder if netcdf people should be alerted, or at least provide an option to suppress this behavior.

…

-Fan From: Nathan Potter <notifications@github.com<mailto:notifications@github.com>> Reply-To: OPENDAP/bes <reply@reply.github.com<mailto:reply@reply.github.com>> Date: Tuesday, February 28, 2017 at 10:23 AM To: OPENDAP/bes <bes@noreply.github.com<mailto:bes@noreply.github.com>> Cc: "Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]" <fan.fang-1@nasa.gov<mailto:fan.fang-1@nasa.gov>>, Author <author@noreply.github.com<mailto:author@noreply.github.com>> Subject: Re: [OPENDAP/bes] 'Phony_dim' in .nc4 downloads of 3IMERGM data from Hyrax 1.13.3 (#38) I don't know much about the phony_dim in the fileout_netcdf response. I'll have to look at that. I do know that the "nc_" prefix has something to do with the netCDF library not supporting file names that begin with a number. This limitation may be old and the sever could change this behavior. Again, I'll have to see. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#38 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK_QwSjvkkXessF28rbZ7INsPVfuyWpsks5rhDvsgaJpZM4MOhT5>.

Answer 3 · 2017-02-28T15:43:14.000Z

Let me clarify: In the case of prepending "nc_" to file names that begin with a digit: Hyrax is doing that. And it;s doing that because, at one time, if Hyrax returned a netCDF file whose name began with a digit the file was unusable by the down stream client tools.

Answer 4 · 2017-02-28T15:44:57.000Z

Can we provide a BES key to suppress that in that case? From: Nathan Potter <notifications@github.com<mailto:notifications@github.com>> Reply-To: OPENDAP/bes <reply@reply.github.com<mailto:reply@reply.github.com>> Date: Tuesday, February 28, 2017 at 10:43 AM To: OPENDAP/bes <bes@noreply.github.com<mailto:bes@noreply.github.com>> Cc: "Fang, Fan (GSFC-610.2)[ADNET SYSTEMS INC]" <fan.fang-1@nasa.gov<mailto:fan.fang-1@nasa.gov>>, Author <author@noreply.github.com<mailto:author@noreply.github.com>> Subject: Re: [OPENDAP/bes] 'Phony_dim' in .nc4 downloads of 3IMERGM data from Hyrax 1.13.3 (#38) Let me clarify: In the case of prepending "nc_" to file names that begin with a digit: Hyrax is doing that. And it;s doing that because, at one time, if Hyrax returned a netCDF file whose name began with a digit the file was unusable by the down stream client tools. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#38 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK_QwTsnGGvg4EDXpLf30p3Nb5wi8FXEks5rhECUgaJpZM4MOhT5>.

Answer 5 · 2017-02-28T21:24:08.000Z

Fan,

Short Answer: Yes, although it's an OLFS thing so the config parameter would be added to olfs.xml
Longer Answer:. Before we make this change I think we should examine the roots of the prefixing behavior. If the conditions that required the prefix are no longer operational in the client software base then I think it is reasonable to simply remove the behavior from the server rather than increasing the configuration complexity.

N

Answer 6 · 2017-03-22T13:07:54.000Z

I don't see this in 1.10.1 server, so suspect it is a library issue. We can trying compiling the fileout netcdf module with whatever netcdf library it was compiled with for 1.10.1 to see if the problem goes away.

Answer 7 · 2017-03-22T13:11:56.000Z

My previous post concerns the "phony_dim" issue. I realized that this thread also discussed the issue prepending "nc_" to filenames starting with a digit, which impacted our TRMM and GPM data. Both are critical issues for us.

Answer 8 · 2017-03-23T18:59:05.000Z

This is a file netCDF module issue since the HDF5 handler correctly generates the dimension names.

Answer 9 · 2017-06-11T17:59:25.000Z

These issues have been fixed in Hyrax-1.13.4 release