Intel OPA support?

Question

Intel OPA support?

Closed this issue 6 years ago · 9 comments

gshamov commented 6 years ago

Hi,

Is Intel OmniPath supported for the recent GA 5.7?
I can build it, but one of the tests keeps failing.

--
Grigory Shamov
University of Manitoba

gshamov commented 6 years ago

Thanks!

Answer 1 · 2018-11-28T23:06:36.000Z

Use ARMCI_NETWORK=MPI-PR with Intel Omni Path.

Answer 2 · 2018-11-29T15:46:38.000Z

Somehow didn't help.

Answer 3 · 2018-11-29T17:03:06.000Z

Please provide full details if you want help debugging.

Answer 4 · 2018-12-04T22:04:37.000Z

Hi Jeff,

Sorry about the delay. Looks like it was a problem with our OpeMPI.: the test perf_strided was having error "Exhausted 1048576 MQ irecv ... PSM2_MQ_RECVREQS_MAX=1048576". I was attempting to compile GA with Intel compilers 2016.4.258 and OpenMPI 2.1.1 . I did the following configure options:

ARMCI_NETWORK=MPI-PR ./configure --enable-ofi --with-mpi-pr CC=icc FC=ifort

When I have tried OpenMPI 3.1.2 also with Intel compilers, and the same configure flags the problem went away and all the tests passed.

Answer 5 · 2018-12-04T22:09:01.000Z

@gshamov Ah, okay, I have seen that one before with NWChem. Try PSM2_MEMORY=large.

It seems the GASNet folks have encountered this error and solution before (https://bitbucket.org/berkeleylab/gasnet/pull-requests/73/fix-for-psm-conduit-puts-reporting/activity#comment-30348715).

Answer 6 · 2018-12-05T17:50:35.000Z

Thanks!
Another question: can GA binaries understand more than one fabric? Like. both OPA and IB, if configured with multiple --enable- flags?

Answer 7 · 2018-12-05T19:15:38.000Z

@gshamov No, GA does not support that, but MPI and OFI do, so if you want a binary that runs over multiple fabrics, configure GA to use MPI-PR or ARMCI-MPI and run with an MPI library that supports IB+OPA. Intel MPI does that automatically, Open-MPI can be built to do it, and both MPICH and Open-MPI can do it via OFI if OFI is configured properly.

Answer 8 · 2018-12-05T19:16:57.000Z

@gshamov Note that this is great feature request but it is a lot of work and is unlikely to be implemented, particularly since it is much easier to improve GA support for MPI and punt the problem to MPI.