ValueError: cannot reshape array of size 360 into shape (1032,)
Opened this issue · 5 comments
This error occurred when I calculated THE PET. I did my best but couldn't solve it, hoping someone would help me.
C:\Users\izzy>process_climate_indices --index pet --periodicity monthly --netcdf_temp Y:/Eurasia_pr/CA/CA/tas/RA_tas_Amon_MIROC6_ssp585_r1i1p1f1_gn_201501_210012.nc --var_name_temp tas --output_file_base Y:/RA_pr/MIROC6_ssp585_2015_2100 --multiprocessing all_but_one
2021-06-02 17:44:25 INFO Start time: 2021-06-02 17:44:25.672740
2021-06-02 17:44:25 INFO Computing PET
2021-06-02 17:47:32 ERROR Failed to complete
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "e:\anaconda3\lib\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "e:\anaconda3\lib\multiprocessing\pool.py", line 48, in mapstar
return list(map(*args))
File "e:\anaconda3\lib\site-packages\climate_indices_main_.py", line 1382, in _apply_along_axis_double
second_np_array = np.frombuffer(second_array.get_obj()).reshape(shape[0])
ValueError: cannot reshape array of size 360 into shape (1032,)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "e:\anaconda3\lib\site-packages\climate_indices_main_.py", line 1714, in main
arguments.netcdf_pet, arguments.var_name_pet = compute_write_index(
File "e:\anaconda3\lib\site-packages\climate_indices_main.py", line 1049, in _compute_write_index
parallel_process(
File "e:\anaconda3\lib\site-packages\climate_indices_main.py", line 1296, in _parallel_process
pool.map(_apply_along_axis_double, chunk_params)
File "e:\anaconda3\lib\multiprocessing\pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "e:\anaconda3\lib\multiprocessing\pool.py", line 771, in get
raise self.value
File "e:\anaconda3\lib\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "e:\anaconda3\lib\multiprocessing\pool.py", line 48, in mapstar
return list(map(*args))
File "e:\anaconda3\lib\site-packages\climate_indices_main.py", line 1382, in apply_along_axis_double
second_np_array = np.frombuffer(second_array.get_obj()).reshape(shape[0])
ValueError: cannot reshape array of size 360 into shape (1032,)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "e:\anaconda3\lib\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "e:\anaconda3\lib\multiprocessing\pool.py", line 48, in mapstar
return list(map(*args))
File "e:\anaconda3\lib\site-packages\climate_indices_main.py", line 1382, in _apply_along_axis_double
second_np_array = np.frombuffer(second_array.get_obj()).reshape(shape[0])
ValueError: cannot reshape array of size 360 into shape (1032,)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "e:\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
return run_code(code, main_globals, None,
File "e:\anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "E:\Anaconda3\Scripts\process_climate_indices.exe_main.py", line 7, in
File "e:\anaconda3\lib\site-packages\climate_indices_main.py", line 1714, in main
arguments.netcdf_pet, arguments.var_name_pet = compute_write_index(
File "e:\anaconda3\lib\site-packages\climate_indices_main.py", line 1049, in _compute_write_index
parallel_process(
File "e:\anaconda3\lib\site-packages\climate_indices_main.py", line 1296, in _parallel_process
pool.map(_apply_along_axis_double, chunk_params)
File "e:\anaconda3\lib\multiprocessing\pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "e:\anaconda3\lib\multiprocessing\pool.py", line 771, in get
raise self.value
File "e:\anaconda3\lib\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "e:\anaconda3\lib\multiprocessing\pool.py", line 48, in mapstar
return list(map(*args))
File "e:\anaconda3\lib\site-packages\climate_indices_main.py", line 1382, in _apply_along_axis_double
second_np_array = np.frombuffer(second_array.get_obj()).reshape(shape[0])
ValueError: cannot reshape array of size 360 into shape (1032,)
I have solved this problem. After I modified the order of the dimensions in the main.py file, I did not modify the index in the following 'shape[ i ]' in time. When I changed it to the correct index of 'lat', the error was resolved.
@Izzylzy can you elaborate a bit more, please? I have the same error but don't understand your solution? Did you modify the source code?
@monocongo any clue what I could try to find a solution? I tried calculating SPI & SPEI and it works without any problems.
According to the error message, it seems there is an issue with the AWC data. What could cause the problem? I use ERA5 reanalysis temperature, precipitation and soil moisture with the same spatial and temporal resolution.
My system runs on Ubuntu 20.04, using Python 3.8 in a fresh conda enviroment. I use climate-indices version 1.0.9.
Input:
process_climate_indices --index palmers --periodicity monthly --netcdf_precip prcp.nc --var_name_precip tp --netcdf_pet pe.nc --var_name_pet pev --netcdf_awc sm.nc --var_name_awc swvl1 --output_file_base output --calibration_start_year 1979 --calibration_end_year 2019 --multiprocessing all
Output:
2021-09-01 12:29:39 INFO Start time: 2021-09-01 12:29:39.649277
2021-09-01 12:29:44 INFO Computing PALMERS
2021-09-01 12:29:46 ERROR Failed to complete
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/site-packages/climate_indices/__main__.py", line 1439, in _apply_along_axis_palmers
awc_np_array = np.frombuffer(awc_array.get_obj()).reshape([shape[0], shape[1]])
ValueError: cannot reshape array of size 32659200 into shape (180,360)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/site-packages/climate_indices/__main__.py", line 1813, in main
_compute_write_index(kwrgs)
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/site-packages/climate_indices/__main__.py", line 859, in _compute_write_index
_parallel_process(
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/site-packages/climate_indices/__main__.py", line 1296, in _parallel_process
pool.map(_apply_along_axis_palmers, chunk_params)
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
ValueError: cannot reshape array of size 32659200 into shape (180,360)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/site-packages/climate_indices/__main__.py", line 1439, in _apply_along_axis_palmers
awc_np_array = np.frombuffer(awc_array.get_obj()).reshape([shape[0], shape[1]])
ValueError: cannot reshape array of size 32659200 into shape (180,360)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/nrieger/anaconda3/envs/climate-indices/bin/process_climate_indices", line 8, in <module>
sys.exit(main())
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/site-packages/climate_indices/__main__.py", line 1813, in main
_compute_write_index(kwrgs)
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/site-packages/climate_indices/__main__.py", line 859, in _compute_write_index
_parallel_process(
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/site-packages/climate_indices/__main__.py", line 1296, in _parallel_process
pool.map(_apply_along_axis_palmers, chunk_params)
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/nrieger/anaconda3/envs/climate-indices/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
ValueError: cannot reshape array of size 32659200 into shape (180,360)
@nicrie this looks to be happening within the guts of the multiprocessing-specific code for simulating shared memory across processes. It looks to have something to do with a mis-shaped variable.
I'd like to point out that the Palmers are the least solid indices in this package. I spent a couple of years off and on reimplementing that code in Python but it was never sufficiently vetted by the scientists I worked with at NOAA, and there are some questions yet as to its validity. My hope was that someone would take the work done here and run with it but apparently, nobody else wants to develop/validate this code. To be frank I don't know why Palmers are even still used -- I've looked under the hood and it's a true Rube Goldberg machine. If you've successfully computed SPI and SPEI using this package then honestly that's as far as I'd take it, the Palmers (PDSI etc.) are not publication-worthy, I should just remove these from the package (or prominently mark them as experimental and not for use in research).
Thanks for sharing your experience, that's good to know! I'll stick with SPI/SPEI then :)