allow namelist control of PIO options

When using PIO, the non-CESM user is limited to a specified configuration of PIO. Either netcdf or pnetcdf is allowed as the iotype, the rearranger is limited to box and the stride and number of iotasks is limited to 4 and nprocs/4, respectively.

Testing in UFS has shown that better IO performance at high PE counts can potentially be achieved using either the subset rearranger and specifying PE dependent stride and iotasks but this requires allowing run-time configuration instead of compile-time configuration of PIO.

I have a working feature branch using an optional new namelist setting pio_options. Currently it allows control of 4 settings (ie, none of the rearranger options) specified as follows:

pio_options = iotype, rearranger type, iostride, numiotasks

I am debugging an issue with reading a restart file using the subset rearranger, although writing a restart using the same options works without issue.

This looks sensible. We have relied on the driver to send the PIO options for CICE.

This sounds good to me too. For CESM, the driver can always overwrite the namelist defined values, and it will be good to provide for additional pio fine tuning in the standalone model. Thanks.

I've been unable to resolve the issue with setting the subset rearranger and reading a restart file. It appears to fail for some combinations of stride + iotasks. It hangs when reading aicen here:

CICE/cicecore/cicedyn/infrastructure/io/io_pio2/ice_restart.F90

Line 741 in 06282a5

call pio_read_darray(File, vardesc, iodesc3d_ncat, work, status)

For now, when the namelist options are used, I've forced the a restart read to use the box rearranger.

https://github.com/DeniseWorthen/CICE/blob/cffe4ece194d88de5fbe5e0cba06e1e08e46009f/cicecore/cicedyn/infrastructure/io/io_pio2/ice_pio.F90#L121-L129

I'm not entirely satisfied, but it may be a necessary compromise to enable UFS to start setting PIO options via namelist.

This has been a challenge for us in CESM. PIO has some setup options for these, but we had avoided them in CICE standalone.

Thank Dave. At this point, I am actually suspecting a platform specific issue. I was not able to reproduce the problem on Gaea-C5, which uses a much later intel/mpi than Hera. I wanted to test across a couple of other RDHPCS platforms to verify though. I know Jim was puzzled by what I reported.

I've verified at this point that there is no issue reading a restart file w/ the subset re-arranger on Gaea-C5, Derecho and Hercules (MSU HPC). I've played a bit with compiler versions on Hera, but haven't arrived at a compiler/mpi combination that works yet. I'd prefer to nail down where exactly the failure is, but I may not be able to. On the other hand, I am at least learning all about building with spack-stack :-)

Hi Denise. How are you going with this? I've been looking at similar things for the ACCESS model.

A few minor thoughts:

We probably want to raise a warning and ignore the value if lcdf64 is set with a netcdf4 type

CICE/cicecore/cicedyn/infrastructure/io/io_pio2/ice_pio.F90

Line 171 in 1cf109b

if (lcdf64) nmode = ior(nmode,PIO_64BIT_OFFSET)

Otherwise we get very ungraceful failure with a 'Abort with message NetCDF: Invalid argument' error. (We could maybe change the flag to mean use 64bit data if datatype is netcdf4)
The history_format and restart_format options might be confusing now? i.e. setting them to pio_pnetcdf when we are using ParallelIO without pnetcdf would be confusing.

@anton-seaice It's been a while since I've had a chance to look at this, but I think you're right. If we allow all of the various pio iotype values, we should ensure that the lcdf64 flag is consistent.

What I've currently done is change the two format namelist variables to be just pio and then use pio_options to set the iotype which can be any of the iotypes (netcdf, pnetcdf, netcdf4c, netcdf4p). I think that addresses your second point?

What I've currently done is change the two format namelist variables to be just pio and then use pio_options to set the iotype which can be any of the iotypes (netcdf, pnetcdf, netcdf4c, netcdf4p). I think that addresses your second point?

I am actually wondering if we need the *_format namelist option at all? Isn't it controlled by which group of io files are included at compile time? Ill need to do some more investigating!

I think it would be useful to consider #914 as well with this issue. I think we should put together a summary of how io (netcdf, pnetcdf, pio, binary, etc) are controlled and the various flags involved (like lcdf64). Then we should create a set of requirements associated with what we want/need (pio control, netcdf3/4, large files support, consistency between build-time and run-time settings, etc). This should go beyond just what any one person needs at the moment. Lets think about all the various controls/flag that are potentially useful. Then we can come up with an implementation plan that hopefully is extensible.

As a starting point, I have created a google document for this, https://docs.google.com/document/d/1yNK1MV2v51MF_P_qz0Mva34ZxbGSwoPnNvXkm5Q_Hs0. Please add comments (I'm sure it's not complete) and I'll edit those comments into the documentation.

Thanks Tony. I added a couple of comments/suggestions. #858 is also related.

Here is what I was thinking for the error handling:

main...ACCESS-NRI:CICE:io_error

It's could be separate from the rest of the changes.