Problem with cube-gen: NetCDF: Unknown file format
Opened this issue · 7 comments
Hello,
when trying to create a new Cube with the cube-gen tool, I am running into the following error.
(esdc) [user@cluster]$ cube-gen mycube burnt_area:dir=/data/cablab-datacube-0.2.4/low-res/data/burnt_area -c cube-low-res.config
CAB-LAB command-line interface, version 0.2.2
Traceback (most recent call last):
File "/home/user/.conda/envs/esdc/bin/cube-gen", line 11, in <module>
load_entry_point('cablab-core', 'console_scripts', 'cube-gen')()
File "/gpfs0/home/user/cablab-core/cablab/cube_gen.py", line 137, in main
cube.update(source_provider)
File "/gpfs0/home/user/cablab-core/cablab/cube.py", line 118, in update
provider.prepare()
File "/gpfs0/home/user/cablab-core/cablab/cube_provider.py", line 149, in prepare
self._source_time_ranges = self.compute_source_time_ranges()
File "/gpfs0/home/user/cablab-core/cablab/providers/burnt_area.py", line 43, in compute_source_time_ranges
dataset = self.dataset_cache.get_dataset(file)
File "/gpfs0/home/user/cablab-core/cablab/util.py", line 159, in get_dataset
dataset = self.open_dataset(real_file)
File "/gpfs0/home/user/cablab-core/cablab/util.py", line 205, in open_dataset
return netCDF4.Dataset(real_file)
File "netCDF4/_netCDF4.pyx", line 1795, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:12271)
RuntimeError: NetCDF: Unknown file format
Thanks in advance for any help!
hi @georgeblck ,
In order to troubleshoot your issue, could you please give me an overview on what you are trying to achieve by running this cube-gen? Also, what kind of input dataset did you use? If you are interested in the cube data itself, we could provide you an access to our FTP server, which has all of already generated cubes.
Hi @hans-permana ,
my goal is to create a cube based on some new data sets with a higher spatial resolution but a much lower temporal resolution. I am trying to understand the cablab-package and the providers so that I can extend them for the new data sets.
So I was playing around with the cube-gen tool, trying to replicate a usage example from your cube_gen.py file:
cube-gen "cablab-datacube/low-res" "burnt_area:dir=cablab-source/BurntArea"
which I extended by the -c option so that it looks like this:
cube-gen mycube burnt_area:dir=/path/to/netcdfs/burnt_area -c cube-low-res.config
mycube is the name/directory/target of the cube and the burnt_area netcdf-files are from your FTP server.
Does this help?
Thank you for a more detailed information, it has helped me to understand and hence to troubleshoot what your problem is.
It seems to me that there are a couple of issues here. First, the NetCDF error seems to have been caused by incompatibility between the writer and the reader. Could you do a conda list and paste the results here? I found it strange because I just tried it using the esdc environment created by using the environment.yml file and I could not replicate this issue.
Anyway, if you manage to solve this issue, there is another issue waiting. Each provider was created based on the source dataset. In the case of Burnt Area, it is the ones from GFED. So if we run the cube-gen with burnt_area provider, it will not generate anything. It simply does not understand about the different data structure. For this, we can create a provider that can re-sample the generated cube data. Another alternative is, of course, to run the same command on BurntArea source data.
Oh right, now I think I understand. The data on your FTP server are not the actual source data?
So the providers are not for reading the data sets when using them inside Python/Julia but for transforming them into cube conform data sets?
Here is the output of cube list:
# packages in environment at /home/user/.conda/envs/esdc:
#
backports 1.0 py35_1 conda-forge
backports.functools_lru_cache 1.4 py35_1 conda-forge
bokeh 0.12.9 py35_0 conda-forge
bottleneck 1.2.1 py35_1 conda-forge
ca-certificates 2017.7.27.1 0 conda-forge
certifi 2017.7.27.1 py35_0 conda-forge
click 6.7 py35_0 conda-forge
cloudpickle 0.4.0 py35_0 conda-forge
coverage 4.4.1 py35_0 conda-forge
curl 7.45.0 0
cycler 0.10.0 py35_0 conda-forge
cyordereddict 1.0.0 py35_1 conda-forge
dask 0.14.1 py35_0 conda-forge
dask-core 0.15.4 py_0 conda-forge
dbus 1.10.22 0 conda-forge
decorator 4.1.2 py35_0 conda-forge
distributed 1.19.3 py35_0 conda-forge
expat 2.2.1 0 conda-forge
fontconfig 2.12.1 4 conda-forge
freetype 2.7 1 conda-forge
gettext 0.19.7 1 conda-forge
glib 2.53.5 0 conda-forge
gridtools 0.3.0 py35_0 forman
gst-plugins-base 1.8.0 0 conda-forge
gstreamer 1.8.0 1 conda-forge
h5netcdf 0.2.2 py35_0 conda-forge
h5py 2.6.0 np111py35_0
hdf5 1.8.15.1 3
heapdict 1.0.0 py35_0 conda-forge
icu 58.1 1 conda-forge
jinja2 2.9.6 py35_0 conda-forge
jpeg 9b 1 conda-forge
libffi 3.2.1 3 conda-forge
libgfortran 3.0.0 1
libiconv 1.14 4 conda-forge
libnetcdf 4.3.3.1 3
libpng 1.6.28 1 conda-forge
libtiff 4.0.6 5 conda-forge
libxcb 1.12 1 conda-forge
libxml2 2.9.3 6 conda-forge
llvmlite 0.13.0 py35_0
locket 0.2.0 py35_1 conda-forge
markupsafe 1.0 py35_0 conda-forge
matplotlib 2.1.0 py35_0 conda-forge
mkl 2017.0.3 0
msgpack-python 0.4.8 py35_0 conda-forge
ncurses 5.9 10 conda-forge
netcdf4 1.2.2 np111py35_0
networkx 2.0 py35_0 conda-forge
numba 0.28.1 np111py35_0
numpy 1.11.3 py35_0
olefile 0.44 py35_0 conda-forge
openssl 1.0.2l 0 conda-forge
pandas 0.19.0 np111py35_0 conda-forge
partd 0.3.8 py35_0 conda-forge
pcre 8.39 0 conda-forge
pillow 4.3.0 py35_0 conda-forge
pip 9.0.1 py35_0 conda-forge
psutil 5.4.0 py35_0 conda-forge
py 1.4.34 py35_0 conda-forge
pyparsing 2.2.0 py35_0 conda-forge
pyqt 5.6.0 py35_4 conda-forge
pytest 3.2.3 py35_0 conda-forge
pytest-cov 2.5.1 py35_0 conda-forge
python 3.5.2 0 conda-forge
python-dateutil 2.5.3 py35_0 conda-forge
pytz 2016.6.1 py35_0 conda-forge
pyyaml 3.12 py35_1 conda-forge
qt 5.6.2 3 conda-forge
readline 6.2 0 conda-forge
scikit-image 0.12.3 np111py35_0 conda-forge
scipy 0.19.0 np111py35_0
setuptools 27.2.0 py35_0 conda-forge
sip 4.18 py35_1 conda-forge
six 1.10.0 py35_0 conda-forge
sortedcontainers 1.5.7 py35_0 conda-forge
sqlite 3.9.2 0 conda-forge
tblib 1.3.2 py35_0 conda-forge
tk 8.5.19 2 conda-forge
toolz 0.8.2 <pip>
toolz 0.8.2 py_2 conda-forge
tornado 4.5.2 py35_0 conda-forge
wheel 0.29.0 py35_0 conda-forge
xarray 0.8.2 py35_0 conda-forge
xorg-libxau 1.0.8 3 conda-forge
xorg-libxdmcp 1.1.2 3 conda-forge
xz 5.0.5 1 conda-forge
yaml 0.1.6 0 conda-forge
zict 0.1.3 py_0 conda-forge
zlib 1.2.8 3 conda-forge
The data on your FTP server are not the actual source data?
Correct, although we also have an FTP server for the source data. However, the availability to these data is at the moment unclear to me, due to some licensing. Maybe @meggart can say more about this.
So the providers are not for reading the data sets when using them inside Python/Julia but for transforming them into cube conform data sets?
Also correct. Here is a link to more information about providers http://cablab.readthedocs.io/en/latest/esdc_prod.html#writing-a-new-provider
If you want to read the generated data, what you need is Cube Access API
Have fun and let me know if you need some more helps!
About the error, I checked your conda list and it looks ok to me. However I noticed that you don't have cablab-core installed in environment, so it is possible that your cablab-core uses dependencies with different versions, especially the netcdf4 (and its corresponding C-library). Could you try installing cablab-core inside the environment?
I installed the cablab-core in the environment; my conda list now looks like this:
# packages in environment at /home/user/.conda/envs/esdc:
#
backports 1.0 py35_1 conda-forge
backports.functools_lru_cache 1.4 py35_1 conda-forge
bokeh 0.12.9 py35_0 conda-forge
bottleneck 1.2.1 py35_1 conda-forge
ca-certificates 2017.7.27.1 0 conda-forge
cablab-core 0.2.2 <pip>
certifi 2017.7.27.1 py35_0 conda-forge
click 6.7 py35_0 conda-forge
cloudpickle 0.4.0 py35_0 conda-forge
coverage 4.4.1 py35_0 conda-forge
curl 7.45.0 0
cycler 0.10.0 py35_0 conda-forge
cyordereddict 1.0.0 py35_1 conda-forge
dask 0.14.1 py35_0 conda-forge
dask-core 0.15.4 py_0 conda-forge
dbus 1.10.22 0 conda-forge
decorator 4.1.2 py35_0 conda-forge
distributed 1.19.3 py35_0 conda-forge
expat 2.2.1 0 conda-forge
fontconfig 2.12.1 4 conda-forge
freetype 2.7 1 conda-forge
gettext 0.19.7 1 conda-forge
glib 2.53.5 0 conda-forge
gridtools 0.3.0 py35_0 forman
gst-plugins-base 1.8.0 0 conda-forge
gstreamer 1.8.0 1 conda-forge
h5netcdf 0.2.2 py35_0 conda-forge
h5py 2.6.0 np111py35_0
hdf5 1.8.15.1 3
heapdict 1.0.0 py35_0 conda-forge
icu 58.1 1 conda-forge
jinja2 2.9.6 py35_0 conda-forge
jpeg 9b 1 conda-forge
libffi 3.2.1 3 conda-forge
libgfortran 3.0.0 1
libiconv 1.14 4 conda-forge
libnetcdf 4.3.3.1 3
libpng 1.6.28 1 conda-forge
libtiff 4.0.6 5 conda-forge
libxcb 1.12 1 conda-forge
libxml2 2.9.3 6 conda-forge
llvmlite 0.13.0 py35_0
locket 0.2.0 py35_1 conda-forge
markupsafe 1.0 py35_0 conda-forge
matplotlib 2.1.0 py35_0 conda-forge
mkl 2017.0.3 0
msgpack-python 0.4.8 py35_0 conda-forge
ncurses 5.9 10 conda-forge
netcdf4 1.2.2 np111py35_0
networkx 2.0 py35_0 conda-forge
numba 0.28.1 np111py35_0
numpy 1.11.3 py35_0
olefile 0.44 py35_0 conda-forge
openssl 1.0.2l 0 conda-forge
pandas 0.19.0 np111py35_0 conda-forge
partd 0.3.8 py35_0 conda-forge
pcre 8.39 0 conda-forge
pillow 4.3.0 py35_0 conda-forge
pip 9.0.1 py35_0 conda-forge
psutil 5.4.0 py35_0 conda-forge
py 1.4.34 py35_0 conda-forge
pyparsing 2.2.0 py35_0 conda-forge
pyqt 5.6.0 py35_4 conda-forge
pytest 3.2.3 py35_0 conda-forge
pytest-cov 2.5.1 py35_0 conda-forge
python 3.5.2 0 conda-forge
python-dateutil 2.5.3 py35_0 conda-forge
pytz 2016.6.1 py35_0 conda-forge
pyyaml 3.12 py35_1 conda-forge
qt 5.6.2 3 conda-forge
readline 6.2 0 conda-forge
scikit-image 0.12.3 np111py35_0 conda-forge
scipy 0.19.0 np111py35_0
setuptools 27.2.0 py35_0 conda-forge
sip 4.18 py35_1 conda-forge
six 1.10.0 py35_0 conda-forge
sortedcontainers 1.5.7 py35_0 conda-forge
sqlite 3.9.2 0 conda-forge
tblib 1.3.2 py35_0 conda-forge
tk 8.5.19 2 conda-forge
toolz 0.8.2 <pip>
toolz 0.8.2 py_2 conda-forge
tornado 4.5.2 py35_0 conda-forge
wheel 0.29.0 py35_0 conda-forge
xarray 0.8.2 py35_0 conda-forge
xorg-libxau 1.0.8 3 conda-forge
xorg-libxdmcp 1.1.2 3 conda-forge
xz 5.0.5 1 conda-forge
yaml 0.1.6 0 conda-forge
zict 0.1.3 py_0 conda-forge
zlib 1.2.8 3 conda-forge
But the error is still the same. I also tried to do the same thing on a nother System and the same error came up.