alan-turing-institute/AIrsenal

Can't pickle class sqlalchemy.orm.session.Session on Windows

Opened this issue · 2 comments

Get the error below on Windows when trying to run run_airsenal_predictions. Seems related to multiprocessing trying and failing to pickle SQLAlchemy objects.

  File "C:\tools\miniconda3\envs\airsenalenv\Scripts\run_airsenal_predictions-script.py", line 11, in <module>
  File "<string>", line 1, in <module>
    load_entry_point('airsenal', 'console_scripts', 'run_airsenal_predictions')()
  File "c:\users\jackr\github\airsenal\airsenal\scripts\fill_predictedscore_table.py", line 156, in main
  File "C:\tools\miniconda3\envs\airsenalenv\lib\multiprocessing\spawn.py", line 105, in spawn_main
    num_thread=args.num_thread)
  File "c:\users\jackr\github\airsenal\airsenal\scripts\fill_predictedscore_table.py", line 109, in make_predictedscore_table
    exitcode = _main(fd)
  File "C:\tools\miniconda3\envs\airsenalenv\lib\multiprocessing\spawn.py", line 115, in _main
    calc_all_predicted_points(gw_range, season, tag,  session)
      File "c:\users\jackr\github\airsenal\airsenal\scripts\fill_predictedscore_table.py", line 91, in calc_all_predicted_points
self = reduction.pickle.load(from_parent)
EOFErrorprocessor.start():
  File "C:\tools\miniconda3\envs\airsenalenv\lib\multiprocessing\process.py", line 112, in start
Ran out of input
    self._popen = self._Popen(self)
  File "C:\tools\miniconda3\envs\airsenalenv\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\tools\miniconda3\envs\airsenalenv\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\tools\miniconda3\envs\airsenalenv\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\tools\miniconda3\envs\airsenalenv\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session

Short-term workaround might be to make sure we don't use anything from multiprocessing if num_thread = 1, and only support single-threaded on Windows.

The discussion here (plus linked issues) might be relevant: pymeasure/pymeasure#12