encountered errors applying CMEMS (zos) data to py-eddy-tracker identification
doge-py opened this issue · 7 comments
doge-py commented
Hello,
I'm trying to apply adt data from cmems GLOBAL_MULTIYEAR_PHY_001_030 to py-eddy-tracker identification, but encounter error as following:
error:
WARNING 2023-07-25 09:49:47,698 grid.__init__ :
We assume pixel position of grid is centered for /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
DEBUG 2023-07-25 09:49:47,698 grid. load_general_features :
Load general feature from /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
DEBUG 2023-07-25 09:49:47,704 grid. bessel_high_filter :
Run filtering with wavelength of 500 km and order of 3 ...
DEBUG 2023-07-25 09:49:47,704 grid. grid :
Load surf_el from /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
Remain 0:00:00.045110 ETA 2023-07-25 09:49:55.191888 current kernel size : (443, 337) Step : 125/126
DEBUG 2023-07-25 09:49:55,194 grid. bessel_high_filter :
Filtering done
INFO 2023-07-25 09:49:55,195 grid.add_uv :
Add u/v variable with stencil method
DEBUG 2023-07-25 09:49:55,195 grid. compute_stencil :
Stencil half width apply : 4
DEBUG 2023-07-25 09:49:55,198 grid. compute_stencil :
Stencil half width apply : 4
INFO 2023-07-25 09:49:55,924 grid.eddy_identification :
We will apply on step a factor to be coherent with grid : 1.000000
/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numpy/lib/function_base.py:4737: UserWarning: Warning: 'partition' will ignore the 'mask' of the MaskedArray.
arr.partition(
DEBUG 2023-07-25 09:49:55,927 grid. eddy_identification :
Levels from -0.160388 to 0.145720
INFO 2023-07-25 09:49:55,927 eddy_feature.__init__ :
Start computing iso lines
DEBUG 2023-07-25 09:49:55,945 eddy_feature. __init__ :
X shape : (251,)
DEBUG 2023-07-25 09:49:55,946 eddy_feature. __init__ :
Y shape : (126,)
DEBUG 2023-07-25 09:49:55,946 eddy_feature. __init__ :
Z shape : (251, 126)
INFO 2023-07-25 09:49:55,946 eddy_feature.__init__ :
Start computing iso lines with 156 levels from -0.162000 to 0.148000 ...
INFO 2023-07-25 09:49:56,153 eddy_feature.__init__ :
Finish computing iso lines
INFO 2023-07-25 09:49:56,169 eddy_feature.__init__ :
Repair 79 closed contours and 0 almost closed contours / 1026 contours
DEBUG 2023-07-25 09:49:56,172 grid. eddy_identification :
doing collection 1, contour value -0.1600, 1 paths
DEBUG 2023-07-25 09:49:56,188 grid. eddy_identification :
doing collection 2, contour value -0.1580, 1 paths
DEBUG 2023-07-25 09:49:56,201 grid. eddy_identification :
doing collection 3, contour value -0.1560, 1 paths
DEBUG 2023-07-25 09:49:56,201 grid. eddy_identification :
doing collection 4, contour value -0.1540, 1 paths
DEBUG 2023-07-25 09:49:56,201 grid. eddy_identification :
doing collection 5, contour value -0.1520, 1 paths
DEBUG 2023-07-25 09:49:56,201 grid. eddy_identification :
doing collection 6, contour value -0.1500, 1 paths
DEBUG 2023-07-25 09:49:56,202 grid. eddy_identification :
doing collection 7, contour value -0.1480, 1 paths
DEBUG 2023-07-25 09:49:56,202 grid. eddy_identification :
doing collection 8, contour value -0.1460, 1 paths
DEBUG 2023-07-25 09:49:56,202 grid. eddy_identification :
doing collection 9, contour value -0.1440, 1 paths
DEBUG 2023-07-25 09:49:56,202 grid. eddy_identification :
doing collection 10, contour value -0.1420, 1 paths
DEBUG 2023-07-25 09:49:56,202 grid. eddy_identification :
doing collection 11, contour value -0.1400, 1 paths
DEBUG 2023-07-25 09:49:56,202 grid. eddy_identification :
doing collection 12, contour value -0.1380, 1 paths
DEBUG 2023-07-25 09:49:56,203 grid. eddy_identification :
doing collection 13, contour value -0.1360, 1 paths
DEBUG 2023-07-25 09:49:56,203 grid. eddy_identification :
doing collection 14, contour value -0.1340, 1 paths
DEBUG 2023-07-25 09:49:56,203 grid. eddy_identification :
doing collection 26, contour value -0.1100, 1 paths
DEBUG 2023-07-25 09:49:56,203 grid. eddy_identification :
doing collection 27, contour value -0.1080, 1 paths
DEBUG 2023-07-25 09:49:56,203 grid. eddy_identification :
doing collection 28, contour value -0.1060, 1 paths
DEBUG 2023-07-25 09:49:56,203 grid. eddy_identification :
doing collection 29, contour value -0.1040, 1 paths
DEBUG 2023-07-25 09:49:56,203 grid. eddy_identification :
doing collection 30, contour value -0.1020, 2 paths
DEBUG 2023-07-25 09:49:56,204 grid. eddy_identification :
doing collection 31, contour value -0.1000, 2 paths
DEBUG 2023-07-25 09:49:56,204 grid. eddy_identification :
doing collection 32, contour value -0.0980, 2 paths
DEBUG 2023-07-25 09:49:56,204 grid. eddy_identification :
doing collection 33, contour value -0.0960, 2 paths
DEBUG 2023-07-25 09:49:56,204 grid. eddy_identification :
doing collection 34, contour value -0.0940, 2 paths
DEBUG 2023-07-25 09:49:56,204 grid. eddy_identification :
doing collection 35, contour value -0.0920, 2 paths
DEBUG 2023-07-25 09:49:56,205 grid. eddy_identification :
doing collection 36, contour value -0.0900, 2 paths
DEBUG 2023-07-25 09:49:56,205 grid. eddy_identification :
doing collection 37, contour value -0.0880, 2 paths
DEBUG 2023-07-25 09:49:56,205 grid. eddy_identification :
doing collection 38, contour value -0.0860, 2 paths
DEBUG 2023-07-25 09:49:56,205 grid. eddy_identification :
doing collection 39, contour value -0.0840, 2 paths
DEBUG 2023-07-25 09:49:56,205 grid. eddy_identification :
doing collection 40, contour value -0.0820, 2 paths
DEBUG 2023-07-25 09:49:56,206 grid. eddy_identification :
doing collection 41, contour value -0.0800, 3 paths
DEBUG 2023-07-25 09:49:56,206 grid. eddy_identification :
doing collection 42, contour value -0.0780, 2 paths
DEBUG 2023-07-25 09:49:56,206 grid. eddy_identification :
doing collection 43, contour value -0.0760, 2 paths
DEBUG 2023-07-25 09:49:56,207 grid. eddy_identification :
doing collection 44, contour value -0.0740, 2 paths
DEBUG 2023-07-25 09:49:56,207 grid. eddy_identification :
doing collection 45, contour value -0.0720, 3 paths
DEBUG 2023-07-25 09:49:56,207 grid. eddy_identification :
doing collection 46, contour value -0.0700, 3 paths
DEBUG 2023-07-25 09:49:56,207 grid. eddy_identification :
doing collection 47, contour value -0.0680, 6 paths
DEBUG 2023-07-25 09:49:56,208 grid. eddy_identification :
doing collection 48, contour value -0.0660, 9 paths
DEBUG 2023-07-25 09:49:56,208 grid. eddy_identification :
doing collection 49, contour value -0.0640, 9 paths
DEBUG 2023-07-25 09:49:56,209 grid. eddy_identification :
doing collection 50, contour value -0.0620, 10 paths
DEBUG 2023-07-25 09:49:56,210 grid. eddy_identification :
doing collection 51, contour value -0.0600, 9 paths
DEBUG 2023-07-25 09:49:56,210 grid. eddy_identification :
doing collection 52, contour value -0.0580, 12 paths
DEBUG 2023-07-25 09:49:56,211 grid. eddy_identification :
doing collection 53, contour value -0.0560, 11 paths
DEBUG 2023-07-25 09:49:56,212 grid. eddy_identification :
doing collection 54, contour value -0.0540, 11 paths
DEBUG 2023-07-25 09:49:56,212 grid. eddy_identification :
doing collection 55, contour value -0.0520, 9 paths
DEBUG 2023-07-25 09:49:56,213 grid. eddy_identification :
doing collection 56, contour value -0.0500, 11 paths
DEBUG 2023-07-25 09:49:56,214 grid. eddy_identification :
doing collection 57, contour value -0.0480, 11 paths
DEBUG 2023-07-25 09:49:56,215 grid. eddy_identification :
doing collection 58, contour value -0.0460, 15 paths
DEBUG 2023-07-25 09:49:56,216 grid. eddy_identification :
doing collection 59, contour value -0.0440, 16 paths
DEBUG 2023-07-25 09:49:56,217 grid. eddy_identification :
doing collection 60, contour value -0.0420, 21 paths
DEBUG 2023-07-25 09:49:56,218 grid. eddy_identification :
doing collection 61, contour value -0.0400, 21 paths
DEBUG 2023-07-25 09:49:56,220 grid. eddy_identification :
doing collection 62, contour value -0.0380, 19 paths
DEBUG 2023-07-25 09:49:56,221 grid. eddy_identification :
doing collection 63, contour value -0.0360, 22 paths
DEBUG 2023-07-25 09:49:56,223 grid. eddy_identification :
doing collection 64, contour value -0.0340, 18 paths
DEBUG 2023-07-25 09:49:56,225 grid. eddy_identification :
doing collection 65, contour value -0.0320, 17 paths
DEBUG 2023-07-25 09:49:56,226 grid. eddy_identification :
doing collection 66, contour value -0.0300, 18 paths
DEBUG 2023-07-25 09:49:56,228 grid. eddy_identification :
doing collection 67, contour value -0.0280, 14 paths
DEBUG 2023-07-25 09:49:56,230 grid. eddy_identification :
doing collection 68, contour value -0.0260, 13 paths
DEBUG 2023-07-25 09:49:56,240 grid. eddy_identification :
doing collection 69, contour value -0.0240, 11 paths
DEBUG 2023-07-25 09:49:56,241 grid. eddy_identification :
doing collection 70, contour value -0.0220, 12 paths
DEBUG 2023-07-25 09:49:56,242 grid. eddy_identification :
doing collection 71, contour value -0.0200, 10 paths
DEBUG 2023-07-25 09:49:56,243 grid. eddy_identification :
doing collection 72, contour value -0.0180, 8 paths
DEBUG 2023-07-25 09:49:56,244 grid. eddy_identification :
doing collection 73, contour value -0.0160, 5 paths
DEBUG 2023-07-25 09:49:56,244 grid. eddy_identification :
doing collection 74, contour value -0.0140, 5 paths
DEBUG 2023-07-25 09:49:56,244 grid. eddy_identification :
doing collection 75, contour value -0.0120, 3 paths
DEBUG 2023-07-25 09:49:56,245 grid. eddy_identification :
doing collection 76, contour value -0.0100, 5 paths
DEBUG 2023-07-25 09:49:56,245 grid. eddy_identification :
doing collection 77, contour value -0.0080, 6 paths
Traceback (most recent call last):
File "/home/user/eddy_identification.py", line 19, in <module>
a, c = h.eddy_identification(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 849, in eddy_identification
) = self.get_uavg(
^^^^^^^^^^^^^^
File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 996, in get_uavg
max_average_speed = self.speed_coef_mean(original_contour)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 1911, in speed_coef_mean
return mean_on_regular_contour(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args
error_rewrite(e, 'typing')
File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite
raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<built-in function getitem>) found for signature:
>>> getitem(bool, UniTuple(int64 x 2))
There are 22 candidate implementations:
- Of which 22 did not match due to:
Overload of function 'getitem': File: <numerous>: Line N/A.
With argument(s): '(bool, UniTuple(int64 x 2))':
No match.
During: typing of intrinsic-call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (280)
File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py", line 280:
def interp2d_bilinear(x_g, y_g, z_g, m_g, x, y):
<source elided>
if not masked:
if m_g[i0, j0] or m_g[i0, j1] or m_g[i1, j0] or m_g[i1, j1]:
^
During: resolving callee type: type(CPUDispatcher(<function interp2d_bilinear at 0x7efed0ffdc60>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (201)
During: resolving callee type: type(CPUDispatcher(<function interp2d_bilinear at 0x7efed0ffdc60>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (201)
File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py", line 201:
def interp2d_geo(x_g, y_g, z_g, m_g, x, y, nearest=False):
<source elided>
else:
return interp2d_bilinear(x_g, y_g, z_g, m_g, x, y)
^
During: resolving callee type: type(CPUDispatcher(<function interp2d_geo at 0x7efed0ffd620>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py (130)
During: resolving callee type: type(CPUDispatcher(<function interp2d_geo at 0x7efed0ffd620>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py (130)
File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 130:
def mean_on_regular_contour(
<source elided>
x_new, y_new = uniform_resample(x_val, y_val, num_fac, fixed_size)
values = interp2d_geo(x_g, y_g, z_g, m_g, x_new[1:], y_new[1:])
my code:
#Activate verbose
from py_eddy_tracker import start_logger
start_logger().setLevel("DEBUG") # Available options: ERROR, WARNING, INFO, DEBUG
#Run identification
from datetime import datetime
from py_eddy_tracker.dataset.grid import RegularGridDataset
grid_name = "mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc"
lon_name = "longitude"
lat_name = "latitude"
h = RegularGridDataset(grid_name, lon_name, lat_name)
h.bessel_high_filter("zos", 500, order=3)
date = datetime(2011, 1, 1)
h.add_uv("zos","ugos","vgos")
a, c = h.eddy_identification(
"zos",
"ugos",
"vgos", # Variables used for identification
date, # Date of identification
0.002, # step between two isolines of detection (m)
pixel_limit=(5, 2000), # Min and max pixel count for valid contour
shape_error=55, # Error max (%) between ratio of circle fit and contour
)
#Save identification data
from netCDF import Dataset
with Dataset(date.strftime("Anticyclonic_%Y%m%d.nc"), "w") as h:
a.to_netcdf(h)
with Dataset(date.strftime("Cyclonic_%Y%m%d.nc"), "w") as h:
c.to_netcdf(h)other information:
numpy version : 1.24.4
numba version : 0.57.1
python version : 3.11.3
Best regards,
doge
doge-py commented
after changing acess data method from MOTU to ftp, got the same errors from here Unit error block EddyId #2.
July 27, 2023
after some research, problem solved when I add force_speed_unit="m/s" to eddy_identification, but I still want to know how to apply data obtained from MOTU because it seems that ftp can't subset data variables? (not quite sure about it)
new code:
#Activate verbose
from py_eddy_tracker import start_logger
start_logger().setLevel("DEBUG") # Available options: ERROR, WARNING, INFO, DEBUG
#Run identification
from datetime import datetime
from py_eddy_tracker.dataset.grid import RegularGridDataset
grid_name = "mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc"
lon_name = "longitude"
lat_name = "latitude"
h = RegularGridDataset(grid_name, lon_name, lat_name)
h.bessel_high_filter("zos", 500, order=3)
date = datetime(2011, 1, 1)
h.add_uv("zos","ugos","vgos")
a, c = h.eddy_identification(
"zos",
"ugos",
"vgos", # Variables used for identification
date, # Date of identification
0.002, # step between two isolines of detection (m)
pixel_limit=(5, 2000), # Min and max pixel count for valid contour
shape_error=55, # Error max (%) between ratio of circle fit and contour
force_speed_unit="m/s"
)
#Save identification data
from netCDF import Dataset
with Dataset(date.strftime("Anticyclonic_%Y%m%d.nc"), "w") as h:
a.to_netcdf(h)
with Dataset(date.strftime("Cyclonic_%Y%m%d.nc"), "w") as h:
c.to_netcdf(h)
Best regards,
doge
AntSimi commented
MOTU do some modifications from original file, last time i check this tools create a mix of nan and masked value.
Maybe you could try to use 'nan_masking' options #111 (comment)
doge-py commented
I followed your suggestions and add "nan_masking=True" to the code, but unfortunately, the issue still persists and got exactly same error message.
I'm thankful for your assistance.
AntSimi commented
could you share your file in order to reproduce bug?
doge-py commented
I subsetted it as following:
time: 2020,12,31 ~ 2020,12,31
lon: 67 ~ 105
lat: -37 ~ -29
elevation: -0.49 ~ -0.49
variable: uo, vo, zos
AntSimi commented
with latest version you must avoid this exception with& without nan_masking option
doge-py commented
It worked; I really appreciate your help!