Unit test failure with ifort 2023.1 on chinook

[kshedstrom]

I thought I would try intel's 2023 compilers to see if I still have some of my optimizer troubles with 2022. Instead, it has spawned some new troubles:

• netcdf failures with the system libraries
• netcdf failure to build the latest netcdf-fortran from source (4.6.1)
• netcdf failure to "make check" cleanly from source for 4.6.0
• all seems well with 4.5.3 (didn't try 4.5.4)

But now I get an error from the unit_tests:

 ==== remapping_attic: remapping_attic_unit_tests =================
 h0 (test data)
i=         1         2         3         4         5
x=  0.00E+00  7.50E-01  1.50E+00  2.25E+00  3.00E+00
i=              1         2         3         4
h=       7.50E-01  7.50E-01  7.50E-01  7.50E-01
u=       9.00E+00  3.00E+00 -3.00E+00 -9.00E+00
 h1 (by delta)
i=         1         2         3         4
x=  0.00E+00  1.00E+00  2.00E+00  3.00E+00
i=              1         2         3
h=       1.00E+00  1.00E+00  1.00E+00
u=       8.00E+00 -1.11E-16 -8.00E+00
 h2
i=         1         2         3         4         5         6         7
x=  0.00E+00  5.00E-01  1.00E+00  1.50E+00  2.00E+00  2.50E+00  3.00E+00
i=              1         2         3         4         5         6
h=       5.00E-01  5.00E-01  5.00E-01  5.00E-01  5.00E-01  5.00E-01
u=       1.00E+01  6.00E+00  2.00E+00 -2.00E+00 -6.00E+00 -1.00E+01
 hn2
i=         1         2         3         4         5         6         7
x=  0.00E+00  5.00E-01  1.00E+00  1.50E+00  2.00E+00  2.50E+00  3.00E+00
i=              1         2         3         4         5         6
h=       5.00E-01  5.00E-01  5.00E-01  5.00E-01  5.00E-01  5.00E-01
u=       1.00E+01  6.00E+00  2.00E+00 -2.00E+00 -6.00E+00 -1.00E+01
 remapping_attic_unit_tests: Failed remapByDeltaZ() 2
 === MOM_remapping: interpolation and reintegration unit tests ===
 ==== remapping_attic: remapping_attic_unit_tests =================
 h0 (test data)
i=         1         2         3         4         5
x=  0.00E+00  7.50E-01  1.50E+00  2.25E+00  3.00E+00
i=              1         2         3         4
h=       7.50E-01  7.50E-01  7.50E-01  7.50E-01
u=       9.00E+00  3.00E+00 -3.00E+00 -9.00E+00
 h1 (by delta)
i=         1         2         3         4
x=  0.00E+00  1.00E+00  2.00E+00  3.00E+00
i=              1         2         3
h=       1.00E+00  1.00E+00  1.00E+00
u=       8.00E+00 -1.11E-16 -8.00E+00
 h2
i=         1         2         3         4         5         6         7
x=  0.00E+00  5.00E-01  1.00E+00  1.50E+00  2.00E+00  2.50E+00  3.00E+00
i=              1         2         3         4         5         6
h=       5.00E-01  5.00E-01  5.00E-01  5.00E-01  5.00E-01  5.00E-01
u=       1.00E+01  6.00E+00  2.00E+00 -2.00E+00 -6.00E+00 -1.00E+01
 hn2
i=         1         2         3         4         5         6         7
x=  0.00E+00  5.00E-01  1.00E+00  1.50E+00  2.00E+00  2.50E+00  3.00E+00
i=              1         2         3         4         5         6
h=       5.00E-01  5.00E-01  5.00E-01  5.00E-01  5.00E-01  5.00E-01
u=       1.00E+01  6.00E+00  2.00E+00 -2.00E+00 -6.00E+00 -1.00E+01
 remapping_attic_unit_tests: Failed remapByDeltaZ() 2
 === MOM_remapping: interpolation and reintegration unit tests ===

[marshallward]

I've tried to replicate this on our machine with Intel 2023:

$ ftn --version
ifort (IFORT) 2021.9.0 20230302
Copyright (C) 1985-2023 Intel Corporation.  All rights reserved.

and using the REPRO flags as defined in the NCRC mkmf template:

FCFLAGS = -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -i4 -r8 -nowarn -sox -traceback -O2 -debug minimal -fp-model source  -I../../deps/include -I/opt/cray/pe/netcdf/4.9.0.3/intel/19.0/include

but no luck. Although it's unlikely to make a difference, I've tried with FMS1 (2019.01.03) and FMS2 (2023.02) just in case.

I tested on AMD EPYC 7H12 whereas you are on Intel(R) Xeon(R) CPU E5-2620 v4, so I suppose there is some difference in hardware, but it's going to be hard to track down.

We still have an Intel machine available, so I will repeat this and see what I find.