LLNL/Caliper

undefined symbol: ompt_start_tool

Closed this issue · 4 comments

adam-sim-dev commented

Hello, I compile Caliper-2.6.0 for mfem and link my own code with mfem. My code compiles fine, but when I run it, there is an error:

symbol lookup error: /opt/intel/XE/compilers_and_libraries_2020.4.304/linux/compiler/lib/intel64_lin/libiomp5.so: undefined symbol: ompt_start_tool

However, the mfem/examples/caliper/ex1p could run successfully.

FYI, I am using Ubuntu 20.04, Intel Parallel Studio XE 2020 Update 4 (mpiicpc, and Intel MPI). I have also tried Intel oneAPI 2021.3, the error still exists.

I compile caliper with the following command line:
cmake -DCMAKE_CXX_COMPILER=mpiicpc -DWITH_MPI=On -DBUILD_SHARED_LIBS=Off -DCMAKE_INSTALL_PREFIX=.

I guess the problem may be related to "ompt", so I try to turn on WITH_OMPT in caliper:
cmake -DCMAKE_CXX_COMPILER=mpiicpc -DWITH_OMPT=On -DWITH_MPI=On -DBUILD_SHARED_LIBS=Off -DCMAKE_INSTALL_PREFIX=.

This time, the compilation of caliper fails with error:

Caliper-2.6.0/src/services/ompt/OmptService.cpp(11): catastrophic error: cannot open source file "ompt.h"
  #include <ompt.h>
                   ^
compilation aborted

Hope the above information could help find where the problem is.

Best,
Adam

daboehme commented

Hi @adam-sim-dev ,

That's interesting. The ompt_start_tool function is indeed related to the OpenMP tools interface (ompt), but programs should of course still work without ompt. Can you try set OMP_TOOL=0 as environment variable when running your program? Does it also happen in a build without Caliper?

One issue could be a compiler mismatch somewhere - are you building all components (Caliper, mfem, your own code) with exactly the same tool chain? For Caliper I usually recommend to specifying the serial and MPI compilers separately, i.e.

cmake \ 
  -DCMAKE_C_COMPILER=`which icc` \ 
  -DCMAKE_CXX_COMPILER=`which icpc` \
  -DWITH_MPI=On \
  -DMPI_C_COMPILER=`which mpiicc` \
  -DMPI_CXX_COMPILER=`which mpiicpc` \
  ..

A compiler mismatch could also explain the missing header error - I'd expect an OMPT-capable compiler to have ompt.h in its default search path. However, I haven't tried this with an Intel toolchain yet.

daboehme commented

Hi @adam-sim-dev ,

I've been looking into this a bit further, and I can reproduce the undefined symbol errors with the Intel toolchain. A workaround is to compile Caliper with -DWITH_GOTCHA=Off. We use Gotcha to dynamically instrument library calls like MPI functions, but it has some incompatibilities with the Intel tool stack that we haven't been able to resolve. Note that without Gotcha Caliper falls back to the PMPI interface for instrumenting MPI, which may have slightly higher overheads. The only other workaround currently is using a different compiler.

The missing ompt.h header was actually a bug in Caliper; the OpenMP 5.0 spec renamed that header file. It's fixed in the repository now if you want to try OMPT.

adam-sim-dev commented

Hi @daboehme

Does it also happen in a build without Caliper?

No, there is no problem without Caliper.

For Caliper I usually recommend to specifying the serial and MPI compilers separately, i.e.

cmake \ 
  -DCMAKE_C_COMPILER=`which icc` \ 
  -DCMAKE_CXX_COMPILER=`which icpc` \
  -DWITH_MPI=On \
  -DMPI_C_COMPILER=`which mpiicc` \
  -DMPI_CXX_COMPILER=`which mpiicpc` \
  ..

I am not so familiar with cmake. I will specify different compilers later when I use Caliper.

adam-sim-dev commented

A workaround is to compile Caliper with -DWITH_GOTCHA=Off. We use Gotcha to dynamically instrument library calls like MPI functions, but it has some incompatibilities with the Intel tool stack that we haven't been able to resolve. Note that without Gotcha Caliper falls back to the PMPI interface for instrumenting MPI, which may have slightly higher overheads.

Okay, I think turn Gotcha off is much easier for me to use Caliper now.

The only other workaround currently is using a different compiler.

I will try that. gcc and g++ are available on my computer, while I use BLAS, LAPACK, ScaLapack in Intel MKL for mfem and PETSc, so swith to gcc will rebuild all the packages.

The missing ompt.h header was actually a bug in Caliper; the OpenMP 5.0 spec renamed that header file. It's fixed in the repository now if you want to try OMPT.

Glad to see that you fixed a bug. Thanks for your help. I will close the issue.