Weird results using point clouds of objects from `YCB`

[xEnVrE]

Hi everyone,

I am very interested in this network and I have modified the scripts command_evaluate.sh, evaluate.py and model_concat_upsa.py in order to be able to do inference on point clouds of objects from YCB dataset.

Specifically, I changed the code in a way such that the point clouds of the two frames can have a different number of points as indicated in the paper. I tried again this code with the evaluation set from FlyingThings3D, that you provided, to verify that performance are unchanged.

However, when I try to feed the network with point clouds (xyz + colors in [0, 1]) from YCB I am getting very weird results.

As an example consider the following two frames involving the Cracker Box object

Basically, the point clouds represent one of the faces of the box after it undergoes a small rotation. The rotation is such that the maximum change between points of the two frames is in the order of ~ 4cm as shown in the figure.

When doing inference, the translation vectors from the frame on the left to the frame on the right are very strange and not compatible with that motion, something like

[[[-0.25203004 -0.04336617 -0.24234553]
  [-0.25667757 -0.05793389 -0.23309565]
  [-0.25017104 -0.03977326 -0.2452542 ]
  ...
  [-0.2619986  -0.07882687 -0.21964237]
  [-0.26357266 -0.0843826  -0.21680456]
  [-0.26159576 -0.07794474 -0.21985136]]]

Do you think the point clouds I am feeding to the network are too different from the data from the FlyingThings3D dataset such that the network is not able to generalize with a model trained on that dataset?

Also, I saw that, within the model model_concat_upsa.py, several radiuses are defined, i.e.

RADIUS1 = 0.5
RADIUS2 = 1.0
RADIUS3 = 2.0
RADIUS4 = 4.0

plus some other hardcoded radius like radius = 5.0, radius = 2.4, radius = 1.2 and radius = 0.6 in the definitions of the layers.

In your experience, are these values critical for the network? In case I need to the re-train the network with point clouds, optical flow and colors from scenes involving small objects, e.g. from YCB dataset, should I care in selecting them carefully or they are general enough?

Please find attached the point clouds (xyz + colors) of the two frames (files frame0.txt and frame1.txt) and the results of the inference (file predicted_val.txt) obtained using the pre-trained model available in this repository.

Thank you for your availability.

frame0.txt
frame1.txt
predicted_val.txt

[xingyul]

Yes. Radius values are critical. You have to ensure that the scale of your objects and/or the density of your point cloud suit radius values.

[KevinYuk]

Hi @xEnVrE ,

Could you please share your inference code which you modify from evaluate code? Is that OK?

Thanks.

[xEnVrE]

Hi @KevinYuk,

here's the code.

[KevinYuk]

Hi @KevinYuk,

here's the code.

Hi @xEnVrE ,

Thanks a lot for that.

[Guptajakala]

@xEnVrE Hi, have you tried it out using more reasonable radius? Does it help?