nvidia_p2p_get_pages() failing with error code -22
UTKRISHTPATESARIA opened this issue · 1 comments
UTKRISHTPATESARIA commented
I am implementing NVIDIA GDS with the following hardware config:
Ubuntu 22.04
NVIDA GeForce RTX 3090
Samsung 980 DC NVMe drive.
GDS was installed successfully, but when I was running their test benchmarks it was failing as below:
./gdsio_verify -f /media/nvme/write-test -d 0 -n 1 -s 1G
warn: error opening log file: Permission denied, logging will be disabled
gpu index :0,file :/media/nvme/write-test, gpu buffer alignment :0, gpu buffer offset :0, gpu devptr offset :0, file offset :0, io_requested :1073741824, io_chunk_size :1073741824, bufregister :true, sync :1, nr ios :1,
fsync :0,
Batch mode: 0
cuFileRead returned error(ret=-1, step_size=1073741824, bytes_left=1073741824)
buffer deregister failed :device pointer lookup failure
Checking dmesg logs found:
nvidia-fs:nvfs_pin_gpu_pages:1292 Error ret -22 invoking nvidia_p2p_get_pages
va_start=0x7f6792900000/va_end=0x7f67929fffff/rounded_size=0x100000/gpu_buf_length=0x100000
Digging up some articles I found that GPU Direct RDMA is supported only for Tesla/Quadro class GPU's. I am curious to know whats preventing RTX 3090 to support this, is it something on the hardware that's missing or some driver module?
KiranModukuri commented
GPU BAR1 space is not exposed for consume grade GPUs like RTX3090.