Crash during sparsity pattern creation
victorapm opened this issue · 4 comments
victorapm commented
This issue arises for compositionalMultiphaseFlow at large scale, e.g., level6 problem
Error message on Frontier:
------------------- TIMESTEP START -------------------
- Time: 00h00m00s (0 s)
- Delta Time: 00h00m01s (1 s)
- Cycle: 0
------------------------------------------------------
Time: 0.00e+00 s, dt: 1 s, Cycle: 0
compflowStatistics (time 1 s): Max phase CFL number: 0.0003955364110866208
compflowStatistics (time 1 s): Max component CFL number: 0.00039553641108662084
free(): corrupted unsorted chunks
free(): corrupted unsorted chunks
free(): corrupted unsorted chunks
MPICH ERROR [Rank 1726] [job id 2002214.0] [Wed Jun 5 23:28:00 2024] [frontier00918] - Abort(1) (rank 1726 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1726
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1726
Received signal 11: Segmentation fault
** StackTrace of 10 frames **
Frame 0: /lib64/libpthread.so.0
Frame 1: geos::DofManager::setSparsityPattern(LvArray::SparsityPattern<long long, int, LvArray::ChaiBuffer>&) const
Frame 2: geos::SolverBase::setupSystem(geos::DomainPartition&, geos::DofManager&, LvArray::CRSMatrix<double, long long, int, LvArray::ChaiBuffer>&, geos::HypreVector&, geos::HypreVector&, bool)
Frame 3: geos::SolverBase::solverStep(double const&, double const&, int, geos::DomainPartition&)
Frame 4: geos::SolverBase::execute(double, double, int, int, double, geos::DomainPartition&)
Frame 5: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&)
Frame 6: geos::EventManager::run(geos::DomainPartition&)
Frame 7: geos::GeosxState::run()
Frame 8: main
Frame 9: __libc_start_main
Frame 10: _start
=====
Inspecting the core file with gdb:
Thread 1 "geosx" received signal SIGSEGV, Segmentation fault.
0x00007fffec0fc424 in LvArray::arrayManipulation::emplace<long long, long long const&> (ptr=0x0, size=<optimized out>, index=0, args=@0x7fffffff0798: 10356060)
at /lustre/orion/geo143/scratch/victorapm/projects/geos-dev/src/coreComponents/LvArray/src/arrayManipulation.hpp:489
489 new ( ptr + index ) T( std::forward< ARGS >( args ) ... );
Thread 1 "geosx" received signal SIGSEGV, Segmentation fault.
0x00007fffec0fc424 in LvArray::arrayManipulation::emplace<long long, long long const&> (ptr=0x0, size=<optimized out>, index=0, args=@0x7fffffff0798: 0)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/arrayManipulation.hpp:489
489 new ( ptr + index ) T( std::forward< ARGS >( args ) ... );
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.31-150300.68.1.x86_64 krb5-debuginfo-1.19.2-150400.3.9.1.x86_64 libatomic1-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libbrotlicommon1-debuginfo-1.0.7-3.3.1.x86_64 libbrotlidec1-debuginfo-1.0.7-3.3.1.x86_64 libcom_err2-debuginfo-1.46.4-150400.3.3.1.x86_64 libcurl4-debuginfo-8.0.1-150400.5.41.1.x86_64 libelf1-debuginfo-0.185-150400.5.3.1.x86_64 libgcc_s1-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libgfortran5-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libidn2-0-debuginfo-2.2.0-3.6.1.x86_64 libjitterentropy3-debuginfo-3.4.0-150000.1.9.1.x86_64 libjson-c3-debuginfo-0.13-3.3.1.x86_64 libkeyutils1-debuginfo-1.6.3-5.6.1.x86_64 libldap-2_4-2-debuginfo-2.4.46-150200.14.17.1.x86_64 libncurses6-debuginfo-6.1-150000.5.20.1.x86_64 libnghttp2-14-debuginfo-1.40.0-150200.17.1.x86_64 libnl3-200-debuginfo-3.3.0-1.29.x86_64 libnuma1-debuginfo-2.0.14.20.g4ee5e0c-150400.1.24.x86_64 libopenssl1_1-debuginfo-1.1.1l-150400.7.63.1.x86_64 libpcre1-debuginfo-8.45-150000.20.13.1.x86_64 libpsl5-debuginfo-0.20.1-150000.3.3.1.x86_64 libselinux1-debuginfo-3.1-150400.1.69.x86_64 libstdc++6-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libyaml-0-2-debuginfo-0.1.7-1.17.x86_64 libz1-debuginfo-1.2.11-150000.3.48.1.x86_64 libzstd1-debuginfo-1.5.0-150400.3.3.1.x86_64
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.31-150300.68.1.x86_64 krb5-debuginfo-1.19.2-150400.3.9.1.x86_64 libatomic1-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libbrotlicommon1-debuginfo-1.0.7-3.3.1.x86_64 libbrotlidec1-debuginfo-1.0.7-3.3.1.x86_64 libcom_err2-debuginfo-1.46.4-150400.3.3.1.x86_64 libcurl4-debuginfo-8.0.1-150400.5.41.1.x86_64 libelf1-debuginfo-0.185-150400.5.3.1.x86_64 libgcc_s1-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libgfortran5-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libidn2-0-debuginfo-2.2.0-3.6.1.x86_64 libjitterentropy3-debuginfo-3.4.0-150000.1.9.1.x86_64 libjson-c3-debuginfo-0.13-3.3.1.x86_64 libkeyutils1-debuginfo-1.6.3-5.6.1.x86_64 libldap-2_4-2-debuginfo-2.4.46-150200.14.17.1.x86_64 libncurses6-debuginfo-6.1-150000.5.20.1.x86_64 libnghttp2-14-debuginfo-1.40.0-150200.17.1.x86_64 libnl3-200-debuginfo-3.3.0-1.29.x86_64 libnuma1-debuginfo-2.0.14.20.g4ee5e0c-150400.1.24.x86_64 libopenssl1_1-debuginfo-1.1.1l-150400.7.63.1.x86_64 libpcre1-debuginfo-8.45-150000.20.13.1.x86_64 libpsl5-debuginfo-0.20.1-150000.3.3.1.x86_64 libselinux1-debuginfo-3.1-150400.1.69.x86_64 libstdc++6-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libyaml-0-2-debuginfo-0.1.7-1.17.x86_64 libz1-debuginfo-1.2.11-150000.3.48.1.x86_64 libzstd1-debuginfo-1.5.0-150400.3.3.1.x86_64
bt
(gdb)
#0 0x00007fffec0fc424 in LvArray::arrayManipulation::emplace<long long, long long const&> (ptr=0x0, size=<optimized out>, index=0, args=@0x7fffffff0798: 0)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/arrayManipulation.hpp:489
#1 LvArray::sortedArrayManipulation::insert<long long, LvArray::SparsityPattern<long long, int, LvArray::ChaiBuffer>::CallBacks> (ptr=<optimized out>,
size=<optimized out>, value=@0x7fffffff0798: 0, callBacks=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/sortedArrayManipulation.hpp:653
#2 0x00007fffecc085c8 in LvArray::ArrayOfSetsView<long long, int, LvArray::ChaiBuffer>::insertIntoSetImpl<LvArray::SparsityPattern<long long, int, LvArray::ChaiBuffer>::CallBacks> (this=0x7fffffff2850, i=0, value=@0x7fffffff0798: 0,
cbacks=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/ArrayOfSetsView.hpp:371
#3 LvArray::SparsityPattern<long long, int, LvArray::ChaiBuffer>::insertNonZero (this=0x7fffffff2850, row=0, col=0)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/SparsityPattern.hpp:327
#4 geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::complete (this=0x7fffffff1c58, stack=..., k=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/SparsityKernelBase.hpp:126
#5 geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}::operator()(int) const (
this=this@entry=0x7fffffff1c58, k=k@entry=0)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/SparsityKernelBase.hpp:165
#6 0x00007fffecc07f3a in RAJA::policy::sequential::forall_impl<RAJA::TypedRangeSegment<int, int>, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}, camp::resources::v1::Host, RAJA::expt::ForallParamPack<> >(camp::resources::v1::Host, RAJA::policy::sequential::seq_exec const&, RAJA::TypedRangeSegment<int, int>&&, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&, RAJA::expt::ForallParamPack<>) (body=..., res=..., iter=...)
at /ccs/home/victorapm/projects/spack/opt/spack/install-sles15-zen3-cce-17.0.0-rocm_5.4.3/raja-2024.02.0/include/RAJA/policy/sequential/forall.hpp:99
#7 RAJA::wrap::forall<camp::resources::v1::Host, RAJA::policy::sequential::seq_exec, RAJA::TypedRangeSegment<int, int>, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}, RAJA::expt::ForallParamPack<>&>(camp::resources::v1::Host, RAJA::policy::sequential::seq_exec&&, RAJA::TypedRangeSegment<int, int>&&, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&, RAJA::expt::ForallParamPack<>&) (loop_body=..., r=..., p=..., c=..., f_params=...)
at /ccs/home/victorapm/projects/spack/opt/spack/install-sles15-zen3-cce-17.0.0-rocm_5.4.3/raja-2024.02.0/include/RAJA/pattern/forall.hpp:163
#8 RAJA::policy_by_value_interface::forall<RAJA::policy::sequential::seq_exec, camp::resources::v1::Host, RAJA::TypedRangeSegment<int, int>, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}>(RAJA::policy::sequential::seq_exec&&, camp::resources::v1::Host, RAJA::TypedRangeSegment<int, int>&&, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&) (params=..., p=..., r=..., c=...)
at /ccs/home/victorapm/projects/spack/opt/spack/install-sles15-zen3-cce-17.0.0-rocm_5.4.3/raja-2024.02.0/include/RAJA/pattern/forall.hpp:537
#9 RAJA::forall<RAJA::policy::sequential::seq_exec, RAJA::TypedRangeSegment<int, int>, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}, camp::resources::v1::Host>(RAJA::TypedRangeSegment<int, int>&&, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&) (args=..., args=...)
at /ccs/home/victorapm/projects/spack/opt/spack/install-sles15-zen3-cce-17.0.0-rocm_5.4.3/raja-2024.02.0/include/RAJA/pattern/forall.hpp:578
#10 geos::forAll<RAJA::policy::sequential::seq_exec, int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}>(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&) (end=<optimized out>, body=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/common/GEOS_RAJA_Interface.hpp:223
#11 geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> > (numElems=numElems@entry=100032, kernelComponent=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/SparsityKernelBase.hpp:157
#12 0x00007fffecba2b46 in geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const::{lambda(auto:1&)#1}::operator()<geos::constitutive::NullModel>(geos::CellElementSubRegion&) const::{lambda(auto:1)#1}::operator()<geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2>(geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2) const (finiteElement=..., this=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/KernelBase.hpp:454
#13 geos::finiteElement::FiniteElementDispatchHandler<geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, geos::finiteElement::H1_Wedge_Lagrange1_Gauss6, geos::finiteElement::H1_Tetrahedron_Lagrange1_Gauss1, geos::finiteElement::H1_Pyramid_Lagrange1_Gauss5, geos::finiteElement::ConformingVirtualElementOrder1<4, 3> >::dispatch3D<geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const::{lambda(auto:1&)#1}::operator()<geos::constitutive::NullModel>(geos::CellElementSubRegion&) const::{lambda(auto:1)#1}>(geos::finiteElement::FiniteElementBase&, geos::CellElementSubRegion&&)
(input=..., lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/FiniteElementDispatch.hpp:168
#14 geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const::{lambda(auto:1&)#1}::operator()<geos::constitutive::NullModel>(geos::CellElementSubRegion&) const (
castedConstitutiveRelation=..., this=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/KernelBase.hpp:430
#15 geos::constitutive::ConstitutivePassThru<geos::constitutive::NullModel>::execute<geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const::{lambda(auto:1&)#1}>(geos::constitutive::ConstitutiveBase&, geos::CellElementSubRegion&&) (constitutiveRelation=..., lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/constitutive/ConstitutivePassThru.hpp:206
#16 geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const (targetRegionIndex=0, elementSubRegion=..., this=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/KernelBase.hpp:415
#17 geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, int, int, geos::ElementRegionBase&, geos::CellElementSubRegion&) const (targetIndex=0, subRegion=..., this=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:567
#18 geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const::{lambda(auto:1&)#1}::operator()<geos::CellElementSubRegion>(geos::CellElementSubRegion&) const (castedSubRegion=..., this=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:718
#19 geos::dataRepository::Group::applyLambdaToContainer<geos::CellElementSubRegion, geos::ElementSubRegionBase, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const::{lambda(auto:1&)#1}>(geos::ElementSubRegionBase&, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const::{lambda(auto:1&)#1}&&) (container=..., lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/dataRepository/Group.hpp:452
#20 geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const (targetIndex=0, elementRegion=..., this=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:716
#21 geos::dataRepository::Group::forSubGroups<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(auto:1&)#1}::operator()<geos::ElementRegionBase>(geos::ElementRegionBase&) const (castedSubGroup=..., this=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/dataRepository/Group.hpp:594
#22 geos::dataRepository::Group::applyLambdaToContainer<geos::ElementRegionBase, geos::dataRepository::Group, geos::dataRepository::Group::forSubGroups<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(auto:1&)#1}>(geos::dataRepository::Group&, geos::dataRepository::Group::forSubGroups<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKer#20 geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const (targetIndex=0, elementRegion=..., this=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:716
#23 geos::dataRepository::Group::forSubGroups<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&) (this=0x7c08560, subGroupKeys=..., lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/dataRepository/Group.hpp:592
#24 geos::ElementRegionManager::forElementRegions<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&) (
this=<optimized out>, targetRegions=..., lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:312
#25 geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&) (this=<optimized out>, targetRegions=...,
lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:706
#26 geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&) (this=<optimized out>,
targetRegions=..., lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:560
nels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(auto:1&)#1}&&)
(container=..., lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/dataRepository/Group.hpp:452
#27 geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> > (mesh=..., targetRegions=...,
finiteElementName=..., constitutiveStringName=..., kernelFactory=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/KernelBase.hpp:387
#28 geos::finiteElement::fillSparsity<geos::CellElementSubRegion, geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> (mesh=...,
targetRegions=..., discretizationName=..., inputDofNumber=...,
rankOffset=10362690, inputSparsityPattern=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/SparsityKernelBase.hpp:296
#29 geos::SolidMechanicsLagrangianFEM::setupSystem(geos::DomainPartition&, geos::DofManager&, LvArray::CRSMatrix<double, long long, int, LvArray::ChaiBuffer>&, geos::HypreVector&, geos::HypreVector&, bool)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&)#1}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&) const (this=0x7fffffff2830, mesh=..., regionNames=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/solidMechanics/SolidMechanicsLagrangianFEM.cpp:997
#30 geos::SolverBase::forDiscretizationOnMeshTargets<geos::SolidMechanicsLagrangianFEM::setupSystem(geos::DomainPartition&, geos::DofManager&, LvArray::CRSMatrix<double, long long, int, LvArray::ChaiBuffer>&, geos::HypreVector&, geos::HypreVector&, bool)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&)#1}>(geos::dataRepository::Group&, geos::SolidMechanicsLagrangianFEM::setupSystem(geos::DomainPartition&, geos::DofManager&, LvArray::CRSMatrix<double, long long, int, LvArray::ChaiBuffer>&, geos::HypreVector&, geos::HypreVector&, bool)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&)#1}&&) const (
this=this@entry=0x7c5d500, meshBodies=..., lambda=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/SolverBase.hpp:723
#31 0x00007fffecba09ba in geos::SolidMechanicsLagrangianFEM::setupSystem (
this=0x7c5d500, domain=..., dofManager=..., localMatrix=..., rhs=...,
solution=..., setSparsity=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/solidMechanics/SolidMechanicsLagrangianFEM.cpp:970
#32 0x00007fffecb98548 in geos::SolidMechanicsLagrangianFEM::solverStep (
this=0x7c5d500, time_n=@0x7fffffff2c20: 0, dt=@0x7fffffff2b50: 1,
cycleNumber=0, domain=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/solidMechanics/SolidMechanicsLagrangianFEM.cpp:486
#33 0x00007fffec2fd6f5 in geos::SolverBase::execute (this=0x7c5d500,
time_n=<error reading variable: That operation is not available on integers of more than 8 bytes.>,
dt=<error reading variable: That operation is not available on integers of more than 8 bytes.>, cycleNumber=0, domain=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/SolverBase.cpp:252
#34 0x00007fffed8a76dc in geos::EventBase::execute (this=0x7c56cd0,
time_n=<error reading variable: That operation is not available on integers of more than 8 bytes.>,
dt=<error reading variable: That operation is not available on integers of more than 8 bytes.>, cycleNumber=0, domain=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/events/EventBase.cpp:235
#35 0x00007fffed8abd67 in geos::EventManager::run (this=0x7cba4c0, domain=...)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/events/EventManager.cpp:195
#36 0x00007fffed8bb9d5 in geos::GeosxState::run (this=0x7fffffff3430)
at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mainInterface/GeosxState.cpp:179
#37 0x000000000020e724 in main (argc=5, argv=<optimized out>)
at PATH_TO_GEOS/projects/geos-dev/src/main/main.cpp:50
corbett5 commented
Just looking at the first few lines of the gdb output it looks like arrayManipulation::emplace is being passed a nullptr. Unfortunately the few frames above this argument is optimized out so you can't tell where it is introduced. Any chance this problem is running out of memory, I imagine malloc returning a nullptr might cause this exact problem.
victorapm commented
Thanks, Ben!
It doesn't seem this problem runs out of memory since the high watermarks for the level5 version of it (coarser by a factor of 2 in each direction) look good:
Umpire DEVICE sum across ranks: 22211.7 G
Umpire DEVICE rank max: 11.4 GB
Umpire DEVICE::0 sum across ranks: 22211.7 G
Umpire DEVICE::0 rank max: 11.4 GB
Umpire HOST sum across ranks: 13830.5 G
Umpire HOST rank max: 6.9 GB
Umpire HYPRE_DEVICE sum across ranks: 12683.7 G
Umpire HYPRE_DEVICE rank max: 6.5 GB
Umpire PINNED sum across ranks: 1173.8 GB
Umpire PINNED rank max: 671.9 MB
i.e., we have 64 GB available on the GPU and use only about 18 GB. But, anyway, it would probably be good to double-check that...
I guess recompiling in full Debug mode and perhaps with -DLVARRAY_BOUNDS_CHECK will help us with more info
corbett5 commented
Yeah after a tad more thought I doubt it's an OOM error since in those cases we usually see the error from Umpire directly. But certainly if it's not terribly slow running in full debug mode will hopefully tell us more.
rrsettgast commented
I think Debug mode will be horrendously slow for this case. Perhaps some more targeted debuting via printf statements are in order here? Maybe start with the insertNonZero and everything "finer" that. It will be a bit of a mess, but you stand a better chance than with a debug run at 1/4 frontier scale.