multiprocess Error

[xiangyi-wang]

非常感谢你和你的项目，我在这里面学到很多。 Thank you for your project, which has taught me a lot.The following problems occurred when I used the enwofost module.I don't know how to modify it.

1. enwofost.py : 277 FutureWarning: arrays to stack must be passed as a "sequence" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.

2. multiprocess module have some problems if name == 'main':
   freeze_support()
   ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
   

3. When I modify the code num_of_ensembles = 10
   ens = enwofost(num_of_ensembles, 'potential')
   if name == 'main':
   ens.Generate_With_Dists_From_Objects(woroot + 'par_prior.csv', crop, soil, site, weather, agromanagement)
   ens.PDF_Image('LAI')
   plt.show
   AttributeError: Can't pickle local object 'enwofost.Generate_With_Dists_From_Objects..multiproc_wofost'

[AlexCornelius]

Hi there,

1. Unfortunately I have not experienced this Warning. Could you give an example of the code you used which produced this warning?

2. I don't use this module with name == 'main': so cannot comment on this. Does the module import properly when you use "from enwofost import enwofost" and can you generate 1 single ensemble e.g. num_of_ensembles = 1.

3. the PDF_Image method does not require you to use plt.show if you are working in an interactive environment like jupyter. But if you are working through the command line and using plt.show(), then I can see why this is experiencing issues. Matplotlib is struggling when handling the enwofost object. There are two workarounds:

   a). edit the PDF_Image method to save the figure as as an image to disk at the end of the plotting code. eg plt.savefig('pdf_image_output.png') on line 760.

   b). the PDF_Image returns the 2D array of the PDF distribution, so you could build the plot yourself. using the return of the method:

ens = enwofost(num_of_ensembles, 'potential')
if name == 'main':
ens.Generate_With_Dists_From_Objects(woroot + 'par_prior.csv', crop, soil, site, weather, agromanagement)
image = ens.PDF_Image('LAI')
plt.imshow(image, origin='lower')
plt.show()

[xiangyi-wang]

This is my code and still reporting errors when i try num_of_ensembles = 1,and from enwofost import enwofost have no error;
if I don't use this module with name == 'main',will multiprocess Error: Traceback (most recent call last):
File "", line 1, in
File "E:\Anaconda\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "E:\Anaconda\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "E:\Anaconda\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "E:\Anaconda\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="mp_main")
File "E:\Anaconda\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "E:\Anaconda\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "E:\Anaconda\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "D:\work\wofost_tools-master\wofost_tools-master\test.py", line 28, in
ens.Generate_With_Dists_From_Objects(woroot + 'par_prior.csv', crop, soil, site, weather, agromanagement)
File "D:\work\wofost_tools-master\wofost_tools-master\enwofost.py", line 165, in Generate_With_Dists_From_Objects
manager = multiprocessing.Manager()
File "E:\Anaconda\lib\multiprocessing\context.py", line 56, in Manager
m.start()
File "E:\Anaconda\lib\multiprocessing\managers.py", line 543, in start
self._process.start()
File "E:\Anaconda\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "E:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "E:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 46, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "E:\Anaconda\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "E:\Anaconda\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

`import numpy as np
import datetime as dt
import matplotlib.pyplot as plt
import copy

import dill
from pcse.fileinput import CABOFileReader
from pcse.models import Wofost71_PP
from pcse.fileinput import CABOWeatherDataProvider
from pcse.base.parameter_providers import ParameterProvider
from pcse.util import WOFOST71SiteDataProvider
from pcse.fileinput import YAMLAgroManagementReader
from enwofost import enwofost
#############################################################
woroot ="D:\work\wofost_tools-master\wofost_tools-master\"
crop = CABOFileReader(woroot+'data/henan_crop_params.CAB')
soil = CABOFileReader(woroot+'data/Hengshui.soil')
site = WOFOST71SiteDataProvider(WAV=100, CO2=360)
parameters = ParameterProvider(crop,soil,site)
weather = CABOWeatherDataProvider(woroot+'data/henan_s01HB')
agromanagement = YAMLAgroManagementReader(woroot+'data/timer.amgt')
parameter_object = ParameterProvider(crop,soil,site)
#############################################################
num_of_ensembles = 1
ens = enwofost(num_of_ensembles, 'potential')
if name == 'main':

ens.Generate_With_Dists_From_Objects(woroot + 'par_prior.csv', crop, soil, site, weather, agromanagement)
image = ens.PDF_Image('LAI')
plt.imshow(image, origin='lower')
plt.show()`

[xiangyi-wang]

I tried to run on Linux system without any problems，Here are the reasons I found on the Internet“Because there is no fork (the mechanism of creating process in Linux operating system) in Windows operating system, this file will be automatically imported to start the child process when it is created, and the whole file will be executed when it is imported. So if process () is written directly in a file, it will create subprocess error infinitely recursively. So you have to protect the part that created the child process by using if__name_=='main'. When importing, it will not run recursively.”

[AlexCornelius]

Thanks for investigating this. I developed this on Linux and had no such problems. I think, if it is run on windows I will either develop and windows friendly multiprocessing version or add a single threading version of running WOFOST to avoid the multiprocessing.

[xiangyi-wang]

and, No problem when I use Generate_With_Dists_From_Objects ,
but, Generate_With_MC_From_Objects：
ens.Generate_With_MC_From_Objects(distribution_file, crop, soil, site, weather, agromanagement)
File "/home/wangxiangyi/wofost_tools-master/enwofost.py", line 516, in Generate_With_MC_From_Objects
distributions = np.load(numpy_repo)['retval'][0]
File "/usr/local/lib/python3.6/dist-packages/numpy/lib/npyio.py", line 451, in load
raise ValueError("Cannot load file containing pickled data "
ValueError: Cannot load file containing pickled data when allow_pickle=False
Generate_With_Dists_From_Scratch：
ens.Generate_With_Dists_From_Scratch(distribution_file, crop, soil, site, weather, agromanagement,central_value = 'absolute')
TypeError: Generate_With_Dists_From_Scratch() got multiple values for argument 'central_value'

I don't quite understand the description of these three methods. - Generate_With_Dists_From_Scratch、Generate_With_Dists_From_Objects、 Generate_With_MC_From_Objects；What is the difference between the from strings of file locations、from wofost objects and from wofost objects and Monte Carlo parameter sets.

[AlexCornelius]

The 'Generate_With_MC_From_Objects' method will not work as I have not uploaded any of the MC results. If you have your own MC results, this method will have to be adapted to fit your scenario.

The 'Generate_With_Dists_From_Scratch' method does not support passing in a 'site' object or string. This error is caused by passing too many argument to the method. Therefore to us this method try this:
ens.Generate_With_Dists_From_Scratch('par_prior.csv',
'data/henan_crop_params.CAB', 'data/Hengshui.soil' ,
'data/henan_s01*', 'data/timer.amgt', central_value = 'absolute')

All the method create ensembles, but how they do it is different.
-From_Objects creates ensembles using established WOFOST objects, so you can edit them in python.
-From_Scratch creates ensembles from strings pointing to the location of WOFOST components saved to disk.
-With_Dists creates ensemble by drawing parameters from a normal distribution parameterized by the 'par_prior.csv' file.
-With_MC generates ensembles by randomly selecting a parameter set from and Monte Carlo calibration. No normal distribution is established.