How to draw fiducials in 2D image?

[CYXYZ]

Hello,

when I try to use img and pred_img to draw projected landmarks in register.py, I find that the landmarks and img are not in the same scale. The specific code and the observed result are as follows:

pred_img, mask = registration()
loss = self.criterion(pred_img, img)
losses.append(loss.item())
times.append(0)
est_pose = registration.get_current_pose()
true_fiducials_2d, pred_fiducials_2d = self.specimen.get_2d_fiducials(idx, est_pose)

Create a new figure and subplots

fig, axs = plt.subplots(1, 2, figsize=(10, 5)) # Adjust figsize as needed

Display the original image

axs[0].imshow(img.squeeze().cpu().numpy(), cmap='gray')
axs[0].axis('off') # Hide axes
axs[0].set_title('Original Image')
axs[0].scatter(
pred_fiducials_2d[0, ..., 0].detach().numpy(),
pred_fiducials_2d[0, ..., 1].detach().numpy(),
marker="x",
c="tab:orange",
)

print("pred_img", pred_img)

Display the registered image

axs[1].imshow(pred_img.detach().cpu().numpy()[0,0], cmap='gray')
axs[1].axis('off') # Hide axes
axs[1].set_title('Registered Image')
axs[1].scatter(
true_fiducials_2d[0, ..., 0].detach().numpy(),
true_fiducials_2d[0, ..., 1].detach().numpy(),
label="True Fiducials",
)
axs[1].scatter(
pred_fiducials_2d[0, ..., 0].detach().numpy(),
pred_fiducials_2d[0, ..., 1].detach().numpy(),
marker="x",
c="tab:orange",
label="Predicted Fiducials",
)
for idx in range(true_fiducials_2d.shape[1]):
axs[1].plot(
[true_fiducials_2d[..., idx, 0].item(), true_fiducials_2d[..., idx, 0].item()],
[true_fiducials_2d[..., idx, 1].item(), true_fiducials_2d[..., idx, 1].item()],
"w--",
)

I tried another approach, and here's the code. It resulted in an error.

true_xray , _ = self.specimen[idx]
pred_xray = self.drr(est_pose)
true_fiducials_2d, pred_fiducials_2d = self.specimen.get_2d_fiducials(idx, est_pose)
       
error:
File "/home/data/cyx/autodl-tmp/DiffPose-main/experiments/deepfluoro/register.py", line 224, in run
pred_xray = self.drr(est_pose)
^^^^^^^^^^^^^^^^^^
File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DRR.forward() missing 2 required positional arguments: 'translation' and 'parameterization'

Can you help me identify the reason?

[eigenvivek]

Hi @CYXYZ , the reason for this is because I've updated DiffDRR after releasing DiffPose.

I think if you install pip install diffdrr==0.3.8, all the code in the DiffPose repository will work.

Alternatively, I am updating all the code in DiffPose to work with the latest version of DiffDRR (the update lives in the branch refactor-se3). I'm aiming to release this before the camera ready deadline for CVPR. You can also try using the code in this branch - I'm 90% confident everything works as intended, still need to test a few more functions.

[CYXYZ]

Dear vivek,

I hope this message finds you well. Firstly, I want to express my gratitude for your prompt response and assistance so far. Your expertise and guidance have been invaluable.

I have been working on visualizing the results of 2D X-ray images with fiducials. While I am confident in the functionality of the code you provided, I encountered a challenge with visualizing the registered results with ground truth fiducials. To address this, I made some modifications to register.py to enable visualization of the registered results with ground truth fiducials.

However, I have noticed an issue with the final output. I suspect that the problem might stem from setting the height of DRR to 256 during training, which results in the scale of pred_img in register.py ranging from 0 to 256. On the other hand, the fiducials still maintain the scale of height = 1536 - 100. Consequently, the registered results are not aligning correctly, as illustrated in the attached image.

I am reaching out to inquire if there is a method to adjust the scale of fiducials to accommodate the scale of DRR effectively.

Your insights and suggestions on resolving this issue would be greatly appreciated.

Thank you once again for your assistance.

Best regards,
cyxyz

[CYXYZ]

The scale of x-ray ranges from 0 to 256, while the scale of fiducials is 1436.

[eigenvivek]

Hi @CYXYZ , no problem at all, I'm more than happy to help! Makes me glad to see that the tool is useful for other people.

So when we downsample the image from 1436 pixels -> 256 pixels, we need to adjust the pixel spacings appropriately.

The subsampling factor is subsample = (1536 - 100) / 256. So dividing your 2D fiducial markers by subsample should rescale them to [0, 256].

[CYXYZ]

Dear vivek,

I wanted to express my gratitude for your guidance and assistance in addressing the challenges I encountered recently. Thanks to your valuable advice, I was able to overcome the obstacles I faced.

I have been closely studying your articles and have found them to be incredibly insightful. I am eager to build upon the knowledge and insights you have shared and to contribute to further advancements in our field.

I am currently working on implementing some improvements based on the methods discussed in your articles. I am excited about the opportunity to innovate and refine existing methodologies.

Thank you once again for your support and mentorship. I look forward to future opportunities to collaborate and exchange ideas.

Warm regards,

cyxyz

[eigenvivek]

Great, glad to hear you solved your problem. Looking forward to reading what you do with this code!