SmokinCaterpillar/pypet

pypet slow when using a queue with a pool for large explorations

Closed this issue · 1 comments

Currently pypet is rather slow when using the queue wrapping in combination with a pool.
The problem is that by default traj.v_full_copy=True is set, i.e. every time the trajectory gets pickled the whole parameter space is pickled as well (when you store stuff, for instance). This in turn yields quadratic time complexity (buh!).

This has already been solved within the develop branch. To avoid this issue in older versions explicitly set traj.v_full_copy=False directly before you call env.f_run(my_job).

However, if using freeze_pool_input this workaround cannot be applied because pypet internally sets traj.v_full_copy=True (it won't in the next version, promised). In this case you need to set traj.v_full_copy=False in every single run (at the beginning of your simulation).

This has been solved in the current version.