Reading subsetted data from the ECCO Data Portal
Opened this issue · 8 comments
I have been trying to look at data from the ECCO Data Portal (e.g https://data.nas.nasa.gov/ecco/data.php?dir=/eccodata/llc_4320/regions). This is possible using MATLAB and readbin.m, but I wanted to see if Xmitgcm could read the data as well. I haven't managed. Maybe I am implementing it incorrectly, or I should just stick to MATLAB for this? Thanks!
I called the function as so:
`import numpy
import xarray as xr
import xmitgcm
aste = {'has_faces': True, 'ny': 1489, 'nx': 2400,'nz':1,
'ny_facets': [450,0,270,180,450],
'pad_before_y': [90,0,0,0,0],
'pad_after_y': [0,0,0,90,90],
'face_facets': [0, 0, 2, 3, 4, 4],
'facet_orders' : ['C', 'C', 'C', 'F', 'F'],
'face_offsets' : [0, 1, 0, 0, 0, 1],
'transpose_face' : [False, False, False,
True, True, True]
}
datadir='path_to_my_datadir'
ds = xmitgcm.open_mdsdataset(datadir, prefix='SIarea',iters=[1],nz=1,nx=2400,ny=1489,nz=1,geometry='llc',read_grid=False,extra_metadata=aste)
print(ds) `
The output gives a dataset, but with Data variable empty, following a warning that the available_diagnostics.log couldn't be found.
See screenshot:
Hi @isgiddy -- thanks for your issue! We have a special module just for reading data from the ecco data portal:
https://xmitgcm.readthedocs.io/en/latest/llcreader.html#ecco-http-data-portal
However, it has not been tested with the regional data. Only the global data.
Thanks! This was really helpful. And my apologies, I realise I had not read the Xmitgcm docs thoroughly.
I can now read in the downloaded data using the following code snipet
from xmitgcm import llcreader
from fsspec.implementations.local import LocalFileSystem
import numpy
import matplotlib.pyplot as plt
import xarray as xr
import dask
fs = LocalFileSystem()
store = llcreader.BaseStore(fs, base_path='path_to_data')
model = llcreader.LLC4320Model(store)
# update attributes
llcreader.LLC4320Model.nx=1140
llcreader.LLC4320Model.nz=34
ds_faces = model.get_dataset(varnames=['Theta'],iter_start=82656,iter_step=1,iter_stop=82657,type='faces',k_chunksize=2)
I am not sure if I am correct in redefining nx as I have done accordingly? There does seem to be something wrong with the gridding.
The prevailing issue is for some files and when attempting to read in different depth levels, and with some of the 2D fields, an Assertion Error is called (different to issue #202, because the file is readable in matlab).
I'm guessing that it has something to do with the gridding. Do you have any insights on how to resolve this?
我一直在尝试查看ECCO数据门户中的数据（例如https://data.nas.nasa.gov/ecco/data.php?dir=/eccodata/llc_4320/regions）。使用MATLAB和readbin.m可以做到这一点，但是我想看看Xmitgcm是否也可以读取数据。我还没管好 也许我执行不正确，还是应该坚持使用MATLAB？谢谢！
我这样称呼该函数：
`import numpy
import xarray as xr
import xmitgcmaste = {'has_faces'：True，'ny'：1489，'nx'：2400，'nz'：1，
'ny_facets'：[450,0,270,180,450]，
'pad_before_y'：[90,0,0,0， 0]，
'pad_after_y'：[0,0,0,90,90]，
'face_facets'：[
0、0、2、3、4、4 ]，'facet_orders'：['C'，'C'， 'C'，'F'，'F']，
'face_offsets'：[0，1，0，0，0，1]，
'transpose_face'：[False，False，False，
True，True，True]
}datadir ='path_to_my_datadir'ds
= xmitgcm.open_mdsdataset（datadir，prefix ='SIarea'，iters = [1]，nz = 1，nx = 2400，ny = 1489，nz = 1，geometry ='llc'，read_grid = False ，extra_metadata = aste）
print（ds）`输出警告给出一个数据集，但是Data变量为_empty_，警告是找不到available_diagnostics.log。
看截图：
How to look at data from the ECCO Data Portal (e.g https://data.nas.nasa.gov/ecco/data.php?dir=/eccodata/llc_4320/regions)thtough matlab?This problem has been bothering me for days. If you can read it directly, you can convert the format without xarray!
I am not sure if I am correct in redefining nx as I have done accordingly? There does seem to be something wrong with the gridding.
@isgiddy - Sorry for the slow reply to your question. Could you share the results of your calculation and why you think there is something wrong with the gridding. I can't see how the code you provided could work. This line, for example
store = llcreader.BaseStore(fs, base_path='path_to_data')
is not pointing to any valid dataset. I would recommend starting from this existing model
xmitgcm/xmitgcm/llcreader/known_models.py
Lines 87 to 96 in f5ee774
How to look at data from the ECCO Data Portal (e.g https://data.nas.nasa.gov/ecco/data.php?dir=/eccodata/llc_4320/regions)thtough matlab?This problem has been bothering me for days. If you can read it directly, you can convert the format without xarray!
Hi @csy0101 -- sorry but we are not able to provide MATLAB support here. You're welcome to read the llcreader code and try to re-implement it using MATLAB. Or you could use LLC reader to create xarray objects, write them to local netCDF files, and then switch to MATLAB.
Hi @isgiddy, I am also trying to read regional MITgcm files (see #253) and am wondering if you had made more progress on this issue, e.g. did you figure out a way to adapt xmitgcm to read the regional files or did you wind up writing your own code to do so (and if so, is it publicly available?)
Thanks,
Kyla
Hi @csy0101 -- sorry but we are not able to provide MATLAB support here. You're welcome to read the llcreader code and try to re-implement it using MATLAB. Or you could use LLC reader to create xarray objects, write them to local netCDF files, and then switch to MATLAB.
Hi @rabernat , I am new to python and mitgcm outputs. I have been trying to use 'llcreader' to access 'LLC4320' data. My first goal is to extract global 'Theta' at one vertical level and just one time step from 'LLC4320', and then 'write them to local netCDF file' (I will use Matlab to do the data processing afterwards). Is there any place that I can find codes to do this kind of job? I do not know how to load the selected LLC4320 'Theta' from 'llcreader' and then write them to a local netCDF file.
Thanks,
Xiaodong
Hi @csy0101 -- sorry but we are not able to provide MATLAB support here. You're welcome to read the llcreader code and try to re-implement it using MATLAB. Or you could use LLC reader to create xarray objects, write them to local netCDF files, and then switch to MATLAB.
Hi @rabernat , I am new to python and mitgcm outputs. I have been trying to use 'llcreader' to access 'LLC4320' data. My first goal is to extract global 'Theta' at one vertical level and just one time step from 'LLC4320', and then 'write them to local netCDF file' (I will use Matlab to do the data processing afterwards). Is there any place that I can find codes to do this kind of job? I do not know how to load the selected LLC4320 'Theta' from 'llcreader' and then write them to a local netCDF file.
Thanks, Xiaodong
Hi @xiaodong, I am also new to the LLC4320 DATA and facing the same difficulties with you. Have you find a way to solve your problem?
Thanks, Yu