MLMG solver issues at high density ratio

[dcoveney]

Hi,

I frequently run into MLMG errors when working with high density ratios (~816:1) in IAMR.

The error message I get is
amrex::Abort::0::MLMG failed !!!

The preceding line above the error message is:
... Projection::level_project() at level 0

If I increase the tolerances, e.g. proj_tol etc, the error just occurs at later time steps.

Are there any recommendations you can make for this? Thanks

[asalmgren]

Linear systems with highly variable coefficients are definitely harder for the solvers. What I would suggest as a first pass is to try increasing the number of relaxations at each level of the V-cycle. Try adding the following two lines to your inputs file: nodal_proj.num_pre_smooth = 10 nodal_proj.num_post_smooth = 10 Note that the default values are 2 so this will noticeably slow things down but if this works, you can try backing these numbers off to see what works for you.

…

On Fri, May 15, 2020 at 8:05 AM dcoveney ***@***.***> wrote: Hi, I frequently run into MLMG errors when working with high density ratios (~816:1) in IAMR. The error message I get is amrex::Abort::0::MLMG failed !!! The preceding line above the error message is: ... Projection::level_project() at level 0 If I increase the tolerances, e.g. proj_tol etc, the error just occurs at later time steps. Are there any recommendations you can make for this? Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#41>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YUWP3NMSYZBFN232WLRRVK5JANCNFSM4NBU3A7A> .

-- Ann Almgren Senior Scientist; CCSE Group Lead

[dcoveney]

Thanks for your prompt reply.

I have tried this, but there is no difference - however, when I run it to a smaller final time (before crashing), when AMReX finalises it says that the two variables nodal_proj.num_pre_smooth and nodal_proj.num_post_smooth were unused PP variables:

Deleting projector
Run time = 359.1000167
Unused ParmParse Variables:
  [TOP]::nodal_proj.num_pre_smooth(nvals = 1)  :: [10]
  [TOP]::nodal_proj.num_post_smooth(nvals = 1)  :: [10]

So I could be missing (or misspelling) something and not actually changing anything? Perhaps I am using an old version and there's a different name? I'm working in the Exec/run2d folder (not working with EB or 3D for this test case).

[mic84]

@dcoveney which version (git branch and commit) are you using?

[dcoveney]

I'm using the master branch, I'm not sure how to get the commit info since I cloned the master branch as a downloaded zip from the main page. The release notes say:

04/01/20
**This is a major update since many branches were merged into development during
the month of March
Introducing complex geometries via EB.
No longer supporting constant mu or RZ options.
Update and add regression tests.
Many updates to be compatible with changes to amrex
Various bugfixes for host and GPU calculations  
Various changes to problem setups

Not sure if this helps or not (apologies I can't find the commit).

Do you think I should just clone the latest master branch?

[asalmgren]

We strongly suggest you run with "develop" branch, it contains the latest bug fixes, etc

…

On Fri, May 15, 2020 at 12:09 PM dcoveney ***@***.***> wrote: I'm using the master branch, I'm not sure how to get the commit info since I cloned the master branch as a downloaded zip from the main page. The release notes say: 04/01/20 **This is a major update since many branches were merged into development during the month of March Introducing complex geometries via EB. No longer supporting constant mu or RZ options. Update and add regression tests. Many updates to be compatible with changes to amrex Various bugfixes for host and GPU calculations Various changes to problem setups Not sure if this helps or not (apologies I can't find the commit). Do you think I should just clone the latest master branch? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YXIMHBMQZBMSLP672TRRWHPBANCNFSM4NBU3A7A> .

-- Ann Almgren Senior Scientist; CCSE Group Lead

[dcoveney]

Ah okay, I wasn't aware of that, apologies.

Now that I am working from the development branch, I can say that your suggestion of increasing the number of relaxations has helped, thank you. However there are varying degrees of success depending on the test case, e.g. when there are severe topological changes it still has an issue.

With this in mind, are there any other things I could try? My cells are roughly square and I'm working at a CFL of 0.1. Is it okay to just keep increasing nodal_proj.num_pre_smooth and nodal_proj.num_post_smooth, if the slow down in run time is acceptable? What are the effects of just increasing the projection tolerance, is there a notable drop in the accuracy of the solution?

[asalmgren]

Unfortunately there aren't any perfect solutions. Things you can do: 1) increase num_pre_smooth and num_post_smooth -- no harm to the solution but slows down the solves 2) raise the tolerance of the solver -- this will improve run time but *may* affect your solution, depending on how sensitive your flow is and what you set the tolerance at 3) use an algebraic multigrid (AMG) solver rather than geometric multigrid (GMG). What I suggest is to try using the hypre solve, for example -- basically you do this by setting the bottom_solver to hypre and don't let the multigrid operator coarsen (so that it will only use the bottom solver). See https://amrex-codes.github.io/amrex/docs_html/LinearSolvers.html#external-solvers for details. Note that you will need to download hypre separately, set a path to it, and use USE_HYPRE = TRUE in your make system. 4) If hypre works but is still more expensive than GMG, you can use a "hybrid" approach, where you coarsen one or two levels with GMG then call hypre as a bottom solver. Take away - - there isn't a one-size-fits all approach, but these are good things to try. p.s. Not sure why CFL is relevant for the solve -- it shouldn't be if you're doing a standard projection method ...

…

On Mon, May 18, 2020 at 1:52 AM dcoveney ***@***.***> wrote: Ah okay, I wasn't aware of that, apologies. Now that I am working from the development branch, I can say that your suggestion of increasing the number of relaxations has helped, thank you. However there are varying degrees of success depending on the test case, e.g. when there are severe topological changes it still has an issue. With this in mind, are there any other things I could try? My cells are roughly square and I'm working at a CFL of 0.1. Is it okay to just keep increasing nodal_proj.num_pre_smooth and nodal_proj.num_post_smooth, if the slow down in run time is acceptable? What are the effects of just increasing the projection tolerance, is there a notable drop in the accuracy of the solution? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YROGXJDKQG7Y5SSV6DRSDZLRANCNFSM4NBU3A7A> .

-- Ann Almgren Senior Scientist; CCSE Group Lead