pypet slow when using a queue with a pool for large explorations
Closed this issue · 1 comments
Currently pypet is rather slow when using the queue wrapping in combination with a pool.
The problem is that by default traj.v_full_copy=True
is set, i.e. every time the trajectory gets pickled the whole parameter space is pickled as well (when you store stuff, for instance). This in turn yields quadratic time complexity (buh!).
This has already been solved within the develop branch. To avoid this issue in older versions explicitly set traj.v_full_copy=False
directly before you call env.f_run(my_job)
.
However, if using freeze_pool_input
this workaround cannot be applied because pypet internally sets traj.v_full_copy=True
(it won't in the next version, promised). In this case you need to set traj.v_full_copy=False
in every single run (at the beginning of your simulation).
This has been solved in the current version.