cuMemcpyHtoD failed!

[LexTran]

Hi, I have tried a few times of using deepdrr, and it has done a great job so far. However, I've been encountered an issue lately when performing drr on MSD dataset (also used in CT Spine 1K)

The error message is as follows:

File "/home/ubuntu/disk1/TLX/datasets/deepDRR.py", line 94, in drrGenerate
    image = projector()
  File "/home/ubuntu/disk1/TLX/envs/bx2s/lib/python3.10/site-packages/deepdrr/projector/projector.py", line 1530, in __call__
    return self.project(*args, **kwargs)
  File "/home/ubuntu/disk1/TLX/envs/bx2s/lib/python3.10/site-packages/deepdrr/projector/projector.py", line 556, in project
    cuda.memcpy_htod(
pycuda._driver.LogicError: cuMemcpyHtoD failed: invalid argument

I'm not sure what causes this problem cuz I'm not very familiar with CUDA programming.

I'm guessing my data is too large maybe? The shape of my test data is 608x512x512.

And the code I used for drr is as below:

def drrGenerate(data_dir, output_dir, rotate, split_sign=".nii"):
    for data in sorted(os.listdir(data_dir)):
        outputname = data.split(split_sign)[0]
        if os.path.isfile(output_dir / f'{outputname}.png'):
            continue
        if os.path.isdir(data_dir+data):
            continue
        patient = deepdrr.Volume.from_nifti(     
            data_dir + "/" + data, use_cached=False, use_thresholding=True
        ) 
        patient.faceup()
        rotate_mat = Rotation.from_euler("xyz", [90,rotate,-90], degrees=True)
        patient.world_from_anatomical = geo.FrameTransform.from_rt(
                rotation=rotate_mat.as_matrix().squeeze(),
            )
        z =  -700 
        
        carm = deepdrr.MobileCArm(
            isocenter=patient.center_in_world+geo.v(0,0,z),
            alpha=0,
            beta=0, 
            degrees=True
        )
        with Projector(
            volume=patient,
            carm=carm,
            step=0.1,  # stepsize along projection ray, measured in voxels
            spectrum="120KV_AL43", # energy spectrum 60KV_AL35 90KV_AL40 120KV_AL43
            photon_count=1000000,
            scatter_num=100000,
            threads=8,
            neglog=True, # apply negative log transform to image (convenient for visualization)
        ) as projector:
            image = projector()

        path = output_dir / f'{outputname}.png'
        image_utils.save(path, image)
        log.info(f"saved example projection image to {path.absolute()}")

My test environment is : Ubuntu 22.04 LTS, with a 3080 12GB GPU, CUDA version 11.4, deepdrr 1.1.2 and SimpleITK 2.2.0

---

Additional information, after giving it a few tries, I found that if I cancel the scatter_num attribute, this problem is solved. But I don't understand why, I have tried to decrease the number, but this problem still showed up even for scatter_num = 100.

[sorenchr2011]

I have exactly the same issue.
cuda.memcpy_htod(
self.index_from_world_gpu, np.array(proj.index_from_world)
)
where self.index_from_world_gpu is allocated as 8 float32's and proj.index_from_world is a 4x3 float32. From an admittedly naive viewpoint, it looks correct this should result in a memory segmentation violation but obviously it works for most people - I am also eager to find a solution - the project seems awesome and can't wait to get it working!
I tried with Cuda 11.6 (Ubuntu 22.04( and 11.1 (22.04 and 20.04) - same result.
Soren

[mathiasunberath]

@benjamindkilleen do you have any pointers?

[mathiasunberath]

one thing I would mention is that scatter at the moment is not really supported as we have shifted our focus to other parts of the project. this is largely because scatter (the way we implemented here) did not considerably help our usecases. This is something that may change in the future, but currently we have no clear timeline.

[benjamindkilleen]

I'm the short term, I recommend to disable scatter. (Set scatter_num=0) Long term, this is a feature we hope to make more robust, but it is not a high priority.

[sorenchr2011]

I'm the short term, I recommend to disable scatter. (Set scatter_num=0) Long term, this is a feature we hope to make more robust, but it is not a high priority.

Thank you so much for the help! Good to know that scatter functionality should be avoided for now and now I can run the examples!
Best,
Soren

[LexTran]

I'm the short term, I recommend to disable scatter. (Set scatter_num=0) Long term, this is a feature we hope to make more robust, but it is not a high priority.

Thanks for your help! I do notice that once I canceled the scatter, it works well. I guess I'll just use it without scatter for a while. Hope your project getting better! : ）