cce/16.0.0 internal compiler error

[BenWibking]

On Frontier, RAJA v2022.10.4 fails to build due to an internal compiler error in the Cray C++ compiler 16.0.0 and ROCm 5.5.1:

[ 10%] Building CXX object exercises/CMakeFiles/atomic-histogram_solution.dir/atomic-histogram_solution.cpp.o
cd /autofs/nccs-svm1_home1/wibking/ascent_frontier/build/raja-v2022.10.4/exercises && /opt/cray/pe/craype/2.7.21/bin/CC -D__HIP_PLATFORM_AMD__=1 -D__HIP_PLATFORM_HCC__=1 -I/autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/include -I/autofs/nccs-svm1_home1/wibking/ascent_frontier/build/raja-v2022.10.4/include -isystem /autofs/nccs-svm1_home1/wibking/ascent_frontier/install/camp-2022.10.1/include -Wall -Wextra  -I/opt/rocm-5.5.1/include -Wno-pass-failed    -O3 -DNDEBUG -fPIE --rocm-path=/opt/rocm-5.5.1 -x hip --offload-arch=gfx90a -std=c++14 -MD -MT exercises/CMakeFiles/atomic-histogram_solution.dir/atomic-histogram_solution.cpp.o -MF CMakeFiles/atomic-histogram_solution.dir/atomic-histogram_solution.cpp.o.d -o CMakeFiles/atomic-histogram_solution.dir/atomic-histogram_solution.cpp.o -c /autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/exercises/atomic-histogram_solution.cpp
clang-16: /home/jenkins/clang/lib/CodeGen/CodeGenModule.h:647: clang::CodeGen::CGOpenMPRuntime& clang::CodeGen::CodeGenModule::getOpenMPRuntime(): Assertion `OpenMPRuntime != nullptr' failed.
PLEASE submit a bug report to Cray and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16 -cc1 -triple amdgcn-amd-amdhsa -mllvm -cray-omp-opt-fork-call -mllvm -cray-omp-parallel-opt-call -mllvm -cray-openmp-rename-outlined -fcray-gpu -flocal-restrict -mllvm -cray-enhanced-asm=1 -fenhanced-asm=1 -mllvm -cray-enhanced-ir=1 -fenhanced-ir=1 -fomp-local-offload-table -ffortran -aux-triple x86_64-unknown-linux-gnu -emit-obj -disable-free -clear-ast-before-backend -main-file-name atomic-histogram_solution.cpp -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility=hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/oclc_isa_version_90a.bc -mlink-builtin-bitcode /opt/rocm-5.5.1/amdgcn/bitcode/oclc_abi_version_400.bc -target-cpu gfx90a -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -gheterogeneous-dwarf -resource-dir /opt/cray/pe/cce/16.0.0/cce-clang/x86_64/lib/clang/16 -dependency-file CMakeFiles/atomic-histogram_solution.dir/atomic-histogram_solution.cpp.o.d -MT exercises/CMakeFiles/atomic-histogram_solution.dir/atomic-histogram_solution.cpp.o -sys-header-deps -internal-isystem /opt/cray/pe/cce/16.0.0/cce-clang/x86_64/lib/clang/16/include/cuda_wrappers -idirafter /opt/rocm-5.5.1/include -include __clang_hip_runtime_wrapper.h -isystem /autofs/nccs-svm1_home1/wibking/ascent_frontier/install/camp-2022.10.1/include -isystem /opt/cray/pe/cce/16.0.0/cce-clang/x86_64/lib/clang/16/include -isystem /opt/cray/pe/cce/16.0.0/cce/x86_64/include/craylibs -D __CRAY_X86_TRENTO -D __CRAY_AMD_GFX90A -D __CRAYXT_COMPUTE_LINUX_TARGET -D __HIP_PLATFORM_AMD__=1 -D __HIP_PLATFORM_HCC__=1 -I /autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/include -I /autofs/nccs-svm1_home1/wibking/ascent_frontier/build/raja-v2022.10.4/include -I /opt/rocm-5.5.1/include -D NDEBUG -I /opt/cray/pe/mpich/8.1.26/ofi/cray/14.0/include -I /opt/cray/pe/libsci/23.05.1.4/CRAY/12.0/x86_64/include -I /opt/cray/pe/dsmml/0.2.2/dsmml//include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/cce/16.0.0/cce-clang/x86_64/lib/clang/16/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/cray/pe/cce/16.0.0/cce-clang/x86_64/lib/clang/16/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -Wall -Wextra -Wno-pass-failed -std=c++14 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/autofs/nccs-svm1_home1/wibking/ascent_frontier/build/raja-v2022.10.4/exercises -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -cuid=ec36a848128b6a83 -fcuda-allow-variadic-functions -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/atomic-histogram_solution-25c849/atomic-histogram_solution-gfx90a.o -x hip /autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/exercises/atomic-histogram_solution.cpp
1.	<eof> parser at end of file
2.	/autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/include/RAJA/pattern/forall.hpp:575:40: instantiating function definition 'RAJA::forall<RAJA::policy::hip::hip_exec<256>, RAJA::TypedRangeSegment<int, int> &, (lambda at /autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/exercises/atomic-histogram_solution.cpp:247:61), camp::resources::Hip>'
3.	/autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/include/RAJA/pattern/forall.hpp:518:1: instantiating function definition 'RAJA::policy_by_value_interface::forall<RAJA::policy::hip::hip_exec<256>, camp::resources::Hip, RAJA::TypedRangeSegment<int, int> &, (lambda at /autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/exercises/atomic-histogram_solution.cpp:247:61)>'
4.	/autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/include/RAJA/pattern/forall.hpp:160:1: instantiating function definition 'RAJA::wrap::forall<camp::resources::Hip, RAJA::policy::hip::hip_exec<256>, RAJA::TypedRangeSegment<int, int> &, (lambda at /autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/exercises/atomic-histogram_solution.cpp:247:61), RAJA::expt::ForallParamPack<> &>'
5.	/autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/include/RAJA/policy/hip/forall.hpp:190:1: instantiating function definition 'RAJA::policy::hip::forall_impl<RAJA::TypedRangeSegment<int, int> &, (lambda at /autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/exercises/atomic-histogram_solution.cpp:247:61), 256UL, false, RAJA::expt::ForallParamPack<>>'
6.	/autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/include/RAJA/policy/hip/forall.hpp:137:10: instantiating function definition 'RAJA::policy::hip::impl::forall_hip_kernel<256UL, RAJA::Iterators::numeric_iterator<int>, (lambda at /autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/exercises/atomic-histogram_solution.cpp:247:61), int>'
7.	/autofs/nccs-svm1_home1/wibking/ascent_frontier/RAJA-v2022.10.4/include/RAJA/policy/hip/forall.hpp:137:10: LLVM IR generation of declaration 'RAJA::policy::hip::impl::forall_hip_kernel'
 #0 0x0000000003b45bc7 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x3b45bc7)
 #1 0x0000000003b42f7b SignalHandler(int) Signals.cpp:0:0
 #2 0x00007f221d6a48bf (/lib64/libpthread.so.0+0x168bf)
 #3 0x00007f221c388cba raise (/lib64/libc.so.6+0x4acba)
 #4 0x00007f221c38a354 abort (/lib64/libc.so.6+0x4c354)
 #5 0x00007f221c380cb9 __assert_fail_base (/lib64/libc.so.6+0x42cb9)
 #6 0x00007f221c380d41 __assert_fail (/lib64/libc.so.6+0x42d41)
 #7 0x000000000409c6c4 clang::CodeGen::CodeGenModule::getMangledName(clang::GlobalDecl) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x409c6c4)
 #8 0x00000000040bafb1 clang::CodeGen::CodeGenModule::EmitGlobal(clang::GlobalDecl) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x40bafb1)
 #9 0x00000000040c4792 clang::CodeGen::CodeGenModule::EmitTopLevelDecl(clang::Decl*) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x40c4792)
#10 0x0000000004f4aa1e (anonymous namespace)::CodeGeneratorImpl::HandleTopLevelDecl(clang::DeclGroupRef) ModuleBuilder.cpp:0:0
#11 0x0000000004f3cace clang::BackendConsumer::HandleTopLevelDecl(clang::DeclGroupRef) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x4f3cace)
#12 0x00000000067be32e clang::Sema::InstantiateFunctionDefinition(clang::SourceLocation, clang::FunctionDecl*, bool, bool, bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67be32e)
#13 0x00000000067bc7ce clang::Sema::PerformPendingInstantiations(bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67bc7ce)
#14 0x00000000067beb97 clang::Sema::InstantiateFunctionDefinition(clang::SourceLocation, clang::FunctionDecl*, bool, bool, bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67beb97)
#15 0x00000000067bc7ce clang::Sema::PerformPendingInstantiations(bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67bc7ce)
#16 0x00000000067beb97 clang::Sema::InstantiateFunctionDefinition(clang::SourceLocation, clang::FunctionDecl*, bool, bool, bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67beb97)
#17 0x00000000067bc7ce clang::Sema::PerformPendingInstantiations(bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67bc7ce)
#18 0x00000000067beb97 clang::Sema::InstantiateFunctionDefinition(clang::SourceLocation, clang::FunctionDecl*, bool, bool, bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67beb97)
#19 0x00000000067bc7ce clang::Sema::PerformPendingInstantiations(bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67bc7ce)
#20 0x00000000067beb97 clang::Sema::InstantiateFunctionDefinition(clang::SourceLocation, clang::FunctionDecl*, bool, bool, bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67beb97)
#21 0x00000000067bc7ce clang::Sema::PerformPendingInstantiations(bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x67bc7ce)
#22 0x0000000005e3949c clang::Sema::ActOnEndOfTranslationUnitFragment(clang::Sema::TUFragmentKind) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x5e3949c)
#23 0x0000000005e39c9d clang::Sema::ActOnEndOfTranslationUnit() (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x5e39c9d)
#24 0x0000000005ce25a4 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x5ce25a4)
#25 0x0000000005cd2681 clang::ParseAST(clang::Sema&, bool, bool) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x5cd2681)
#26 0x00000000047afa78 clang::FrontendAction::Execute() (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x47afa78)
#27 0x00000000047284f5 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x47284f5)
#28 0x000000000487fbda clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0x487fbda)
#29 0x0000000000f01b79 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0xf01b79)
#30 0x0000000000efd0d5 ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) driver.cpp:0:0
#31 0x0000000000efdfee clang_main(int, char**) (/opt/cray/pe/cce/16.0.0/cce-clang/x86_64/bin/clang-16+0xefdfee)
#32 0x00007f221c37329c __libc_start_main (/lib64/libc.so.6+0x3529c)
#33 0x0000000000ef6559 _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120:0
clang-16: error: unable to execute command: Aborted (core dumped)
clang-16: error: clang frontend command failed due to signal (use -v to see invocation)
Cray clang version 16.0.0  (5ae52089d93a0a214b7e3ffd5faa680404586e58)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/16.0.0/cce-clang/x86_64/share/../bin
clang-16: note: diagnostic msg: Error generating preprocessed source(s).
gmake[2]: *** [exercises/CMakeFiles/atomic-histogram_solution.dir/build.make:79: exercises/CMakeFiles/atomic-histogram_solution.dir/atomic-histogram_solution.cpp.o] Error 1
gmake[2]: Leaving directory '/autofs/nccs-svm1_home1/wibking/ascent_frontier/build/raja-v2022.10.4'
gmake[1]: *** [CMakeFiles/Makefile2:367: exercises/CMakeFiles/atomic-histogram_solution.dir/all] Error 2
gmake[1]: Leaving directory '/autofs/nccs-svm1_home1/wibking/ascent_frontier/build/raja-v2022.10.4'
gmake: *** [Makefile:139: all] Error 2

Is this a known issue?

[rhornung67]

Not a known issue for us on our MI250x systems. It looks like the error arises during the compilation of one of the RAJA tutorial exercise codes (atomic_histogram_solution). My guess is that you don't need to build those. Please try passing the option -DRAJA_ENABLE_EXERCISES=Off to CMake when you configure RAJA.

Also, trying a newer release of RAJA may help, such as v2022.10.5, v2023.06.0, or v2023.06.1

If you try any of these suggestions, please let us know how it works out.

[BenWibking]

Not a known issue for us on our MI250x systems. It looks like the error arises during the compilation of one of the RAJA tutorial exercise codes (atomic_histogram_solution). My guess is that you don't need to build those. Please try passing the option -DRAJA_ENABLE_EXERCISES=Off to CMake when you configure RAJA.

Adding this option worked around the issue. Thanks.