Realm: invalid argument to cuMemHostGetDevicePointer_v2
Opened this issue · 0 comments
syamajala commented
I'm running S3D through cuda-memcheck and I'm seeing the following error before the main task even starts:
========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid argument" on CUDA API call to cuMemHostGetDevicePointer_v2.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2c60b3]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/realm/cuda/cuda_module.cc:5666:Realm::Cuda::GPUAllocation::register_allocation(Realm::Cuda::GPU*, void*, unsigned long, bool) [0x1238be1]
========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/librealm.so.1
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/realm/cuda/cuda_module.cc:4927:Realm::Cuda::CudaModule::create_dma_channels(Realm::RuntimeImpl*) [0x12347ea]
========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/librealm.so.1
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/realm/runtime_impl.cc:2149:Realm::RuntimeImpl::finish_configure() [0x118ff21]
========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/librealm.so.1
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/realm/runtime_impl.cc:489:Realm::Runtime::finish_configure() [0x11873d2]
========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/librealm.so.1
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/legion/runtime.cc:30212:Legion::Internal::Runtime::start(int, char**, bool, bool, bool) [0x31c1998]
========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/liblegion.so.1
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/legion/legion.cc:7041:Legion::Runtime::start(int, char**, bool, bool, bool) [0x2caa8fc]
========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/liblegion.so.1
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/rhst/s3d_rank_mpi.cc:280:S3DRank::start_legion() [0xa2da4]
========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/librhsf.so
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/rhst/rhst_fortran.cc:133:initialize_rhsf_legion_ [0xa1246]
========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/librhsf.so
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/s3d/source/drivers/solve_driver.f90:196:solve_driver_ [0x1ddaee]
========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/s3d.x
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/s3d/source/drivers/main.f90:131:MAIN__ [0x1dd3e8]
========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/s3d.x
========= Host Frame:/scratch2/seshu/legion_s3d_tdb/s3d/source/drivers/main.f90:8:main [0x682f]
========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/s3d.x
========= Host Frame:../csu/libc-start.c:342:__libc_start_main [0x24083]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame:_start [0x686e]
========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/s3d.x