Indefinite preconditioner when using GAMG for CG
Closed this issue · 4 comments
At some point during the simulation the Poisson solver diverges when using GAMG as a preocnditioner for CG, causing the simulation to stop.
The reason of the divergence is -8, which according to PETSc's documentation is caused by KSP_DIVERGED_INDEFINITE_PC, i.e. the preconditioner is indefinite.
@anushkrish , you have been reporting this issue to PETSC's developer last November (here).
This happens in the "middle" of the simulation, not in every case. It happens to me when running the 3d cylinder case at Re=100.
The fact that it happens somewhere during the simulation is really disturbing. The Poisson matrix is not being changed from one time-step to another. Therefore, the preconditioner should only be computed at the initial time-step.
@anushkrish :have you been investigating this issue a little bit more?
Check either the initial guesses input to the pc or trawl through the pc to find out exactly what conditions cause that error to be thrown. Imagine passing in NaN as one of the initial guesses - what error would petsc give?
I think it's mentioned later in the thread that GAMG can produce preconditioners that are not symmetric positive-definite, and therefore might fail with CG. You should try solving the system using BiCGStab (-ksp2_type bcgs). There are some other suggestions on that thread regarding the GAMG options you can play around with.
Another thing that was brought to my attention was that PetIBM is currently computing the preconditioner at every time step even though the matrix does not change. The function KSPSetReusePreconditioner() was introduced in PETSc 3.5, and it needs to be called if we want to avoid that. We didn't have this problem earlier because this option was set via KSPSetOperators. This will speed up the simulations, but I don't know if it will have any effect on the indefinite preconditioner issue because we are presumably using the same matrix at every time step.
According to this thread, he preconditioner is re-generated before every solve when using GAMG, which is a waste of time for us, since the matrix does not change and the grid remains fixed during the entire simulation.
As Anush mentioned, we could use the function KSPSetReusePreconditioner().
This setting could actually be used for both solvers (Poisson and intermediate fluxes).
BiCGStab can be a workaround for the indefinite preconditioner problem.
Seems this is not a bug in our code. I'm going to remove the bug label.