AMReX MLMG bottom solve change causes PeleLMeX EB_PipeFlow test to fail
baperry2 opened this issue · 4 comments
After AMReX-Codes/amrex#3991 (stemming from Exawind/amr-wind#886) the PeleLMeX EB_PipeFlow test is failing on the nodal projection on the first timestep: https://github.com/baperry2/PeleLMeX/actions/runs/9784946279/job/27016932023.
Despite the CI failing on this test, I've tried with a few different machines and compilers and haven't been able to reproduce the failure locally. However, turning up verbosity for the nodal projection when running locally, I see that since the change the case takes more iterations for the nodal projection to converge (10 vs 8) and there are more frequent bottom solve failures.
@marchdf @WeiqunZhang @asalmgren @jrood-nrel it looks like you all worked the issue in amr-wind. Any idea what might be happening here in LMeX?
Here's what I get running locally:
With amrex 36f111b2c (old):
==================== NEW TIME STEP ====================
Est. time step - Conv: 1.208751229e-05, divu: 1e+200
STEP [0] - Time: 0, dt 1.208751229e-05
- Advance()::MACProjection() --> Time: 0.114974
Nodal Projection:
>> Before projection:
* On lev 0 max(abs(rhs)) = 739.1881134
MLMG: # of AMR levels: 1
# of MG levels on the coarsest AMR level: 3
MLMG: Initial rhs = 739.1881134
MLMG: Initial residual (resid0) = 739.1881134
MLMG: Bottom solve failed.
MLMG: Iteration 1 Fine resid/bnorm = 0.00144668188
MLMG: Iteration 2 Fine resid/bnorm = 4.887726517e-05
MLMG: Iteration 3 Fine resid/bnorm = 2.942642069e-06
MLMG: Iteration 4 Fine resid/bnorm = 1.881492075e-07
MLMG: Iteration 5 Fine resid/bnorm = 1.168997847e-08
MLMG: Iteration 6 Fine resid/bnorm = 7.832919149e-10
MLMG: Iteration 7 Fine resid/bnorm = 5.112867833e-11
MLMG: Iteration 8 Fine resid/bnorm = 3.276700523e-12
MLMG: Final Iter. 8 resid, resid/bnorm = 2.422098078e-09, 3.276700523e-12
MLMG: Timers: Solve = 0.223404125 Iter = 0.200380083 Bottom = 0.014959123
>> After projection:
* On lev 0 max(abs(rhs)) = 125.2587278
- Advance()::VelocityAdvance --> Time: 0.469178
>> PeleLMeX::Advance() --> Time: 0.588156
With amrex 0e3e39b5e (new):
==================== NEW TIME STEP ====================
Est. time step - Conv: 1.208751229e-05, divu: 1e+200
STEP [0] - Time: 0, dt 1.208751229e-05
- Advance()::MACProjection() --> Time: 0.116781
Nodal Projection:
>> Before projection:
* On lev 0 max(abs(rhs)) = 739.1881134
MLMG: # of AMR levels: 1
# of MG levels on the coarsest AMR level: 3
MLMG: Initial rhs = 739.1881134
MLMG: Initial residual (resid0) = 739.1881134
MLMG: Bottom solve failed.
MLMG: Iteration 1 Fine resid/bnorm = 0.001387041191
MLMG: Bottom solve failed.
MLMG: Iteration 2 Fine resid/bnorm = 0.004341203704
MLMG: Bottom solve failed.
MLMG: Iteration 3 Fine resid/bnorm = 9.172997472e-05
MLMG: Bottom solve failed.
MLMG: Iteration 4 Fine resid/bnorm = 0.0001001601963
MLMG: Bottom solve failed.
MLMG: Iteration 5 Fine resid/bnorm = 2.247555741e-06
MLMG: Bottom solve failed.
MLMG: Iteration 6 Fine resid/bnorm = 1.879934597e-07
MLMG: Bottom solve failed.
MLMG: Iteration 7 Fine resid/bnorm = 6.093181729e-09
MLMG: Bottom solve failed.
MLMG: Iteration 8 Fine resid/bnorm = 3.940754528e-10
MLMG: Bottom solve failed.
MLMG: Iteration 9 Fine resid/bnorm = 2.055977701e-11
MLMG: Bottom solve failed.
MLMG: Iteration 10 Fine resid/bnorm = 1.157841855e-12
MLMG: Final Iter. 10 resid, resid/bnorm = 8.558629361e-10, 1.157841855e-12
MLMG: Timers: Solve = 0.316350708 Iter = 0.292605167 Bottom = 0.057702499
>> After projection:
* On lev 0 max(abs(rhs)) = 125.2587278
- Advance()::VelocityAdvance --> Time: 0.567521
>> PeleLMeX::Advance() --> Time: 0.688364
Could you try AMReX-Codes/amrex#4020?
I was able to reproduce the failure with the CI test. And the amrex pr fixes it.