P3M Particle-Mesh solver crashes in scatter
Closed this issue · 9 comments
In GitLab by @mayani_s on Jun 6, 2023, 09:24
The "P3MSolver" class is a Particle-Mesh solver implemented following the implementation of Ulmer in the ippl-orig program p3m3dHeating.cpp to compute space charge forces. However, when using the solver in physics test cases, it causes a crash in the scatter (also serially).
In GitLab by @aaadelmann on Jun 6, 2023, 14:55
Can you provide the name of the branch and the command line i.e. parameters you use for the run?
In GitLab by @mayani_s on Jun 6, 2023, 15:06
I run this on one node (which are the parameters I used for the Open BCs simulation, as suggested by Sri):
srun ./PenningTrap 32 32 32 655360 100 P3M 0.05 --overallocate 1.0 --info 5
You can run this in the master branch, as the P3M solver was merged already before (the only change made was that one can call it in alpine now).
In GitLab by @aaadelmann on Jun 6, 2023, 15:53
the problem comes from
PenningTrap`int ippl::detail::scatterToPoint< ....
and then in Kokkos::atomic_add(double*, desul::Impl::dont_deduce_this_parameter::type) and further down disaster strikes.
In GitLab by @aaadelmann on Jun 7, 2023, 14:21
have you checked max/min(R) before the scatter with different solvers?
In GitLab by @mayani_s on Jun 7, 2023, 14:36
Yes, it seems that the particles do indeed leave the domain. I have found out that the potential computed by the P3M solver is wrong, which is causing the E-field to be too large so particles leave the domain. I have to go back to cross-checking with the ippl-orig p3m3dHeating.cpp solver. I'm thinking of replacing the initial conditions in ippl-orig p3m3dHeating.cpp with the Penning Trap initial conditions to see whether there also the potential is calculated wrong (which would indicate that this Particle-Mesh solver itself is not suited to the problem). If it's right, there is something wrong with my implementation.
In GitLab by @aaadelmann on Jun 7, 2023, 15:54
ok then we have a good lead on that!
In GitLab by @mayani_s on Jun 12, 2023, 10:07
created merge request !187 to address this issue
In GitLab by @mayani_s on Jun 16, 2023, 12:05
After copying in the Penning Trap initial conditions to the ippl-orig branch and combining it with the particle-mesh solver written by Ulmer (calculateGridForces()), it seems like this solver itself shows a discrepancy with the FFT Periodic Poisson Solver in IPPL 2.0. I have attached some slides which show that with a similar source input, the calculated potential is orders of magnitude different.
As the P3MSolver class in IPPL 2.0 is implemented following Ulmer's particle-mesh solver, it makes sense that it was not coinciding in behaviour with the FFT Periodic Poisson solver.
Nevertheless, it should give the same result as the periodic solver. I am not so sure whether we should continue in this direction, trying to find out if there is something wrong with the particle-mesh solver originally implemented in p3m3dHeating.cpp?
If anyone is interested in the test case, it can be found in the branch ippl-orig_sonali in /test/particle/p3mPenning.cpp.