bad_alloc error in function pyEXP.basis.IntegrateOrbits
Closed this issue · 6 comments
I have an array of shape (5165441, 6) containing the phase-space vector [x,y,z,vx,vy,vz] of 5165441 particles.
I am interested in obtaining the orbits of these particles using the function:
times, orbits = pyEXP.basis.IntegrateOrbits(initial_time, final_time, time_step, array, model, pyEXP.basis.AllTimeAccel()),
where 'model' is a list that contains all the basis and coefficients of the BFE.
When I run it, it returns the error: std::bad_alloc. It has something to do with the number of particles used, because for 500k instead of 5M seems to work perfectly fine. It seems to me that there is a limit on the number of particles. Which limit is it? Is there a way to calculate these orbits without having to repeat the 500k-particles-process N times until all particles are used?
Thanks!
The limit is memory on the machine. IntegrateOrbits
will allocate a 3-dimensional array (i.e. rank-3 tensor) which is the rank of the input phase-space array times the number of time steps. So if you asked for 100 time steps, the memory required would be 5165441*6*8*100=24GB
where the factor of 8 is for the double-precision length.
So there are only two solutions:
- Use a machine with more memory
- Do the orbits in batches as you suggested. But even in this case. You will not be able to keep all of the orbits in memory at the same time if you've exceeded the physical memory limits.
Another comment on this: IntegrateOrbits
relies on the user to select an appropriate time step and returns all steps. We could easily add a stride
parameter; that is, return every stride
time step to the user. This would reduce the memory footprint.
I wonder if there is a way to throw a more user-friendly error? Or is that just overkill?
We can easily catch the std::bad_alloc
and provide a user message (from BasisFactory
the C++ side) at the very least. At least the user gets an indicator of the problem. It's also easy to add a limiter on the number of output steps. E.g. perhaps a user doesn't want the entire number of (tfinal-tinit)/h
output steps? We could add an optional nout
parameter which provides the nearest (tfinal-tinit)/nout
times.
There's a new branch in EXP
called IntegrateOrbitMemory
that implements these draft changes. Please post if you try this...
This has been solved in EXP -- recompile for the updated catch error!