Shelves intersect with bowls and bottles in the demo data

[zzilch]

I visualize the demo data and find the shelves intersect with bowls and bottles? Are they supposed to be like this?

Is the demonstration in the teaser available? I think the rack in the teaser is more complicate than the rack in the simulation demos.

[kirby516]

I too am having trouble with intersecting objects. The gripper is colliding with the mug.
Can anyone tell me what is the right answer?

[anthonysimeonov]

Apologies to all for leaving this unanswered this long! OP also reached out over email and I helped them over there, but forgot to update the response to this issue in case this causes others similar trouble. I'm copying my response to @zzilch here for anyone else who is confused about this.

RE shelf intersecting bowls/bottles:
I took a look at the demos and I see what you're talking about -- the sampled shelf point cloud and the query point cloud that we actually use in the optimization ('shelf_pointcloud_uniform' in the .npz file) are both intersecting the bowl/bottle point cloud. Thank you for pointing this out, I didn't catch it when I was collecting the demonstrations. The reason this happened is that in the URDF that I use for the environment, I load the shelf model as a link to the table and I scaled it down a little bit there so it wasn't so tall (this can be seen here). But I didn't apply that same scaling when I sampled the shelf point cloud / query points in the bounding box (which is done in Lines 130-134 in the teleop.py script). This is what is leading the transformed object point cloud in its placement configuration to be lower with respect to the shelf points that are in the demonstrations.

We actually ran our quantitative experiments with these demos, so it seems that this doesn't affect the performance too much. I'm guessing it's because even with the points intersecting the larger full-scale shelf, a large fraction of the query points used in the optimization are still below the bottom surface of bowls/bottles (which leads them to still be approximately aligned via their bottom surface). But perhaps it can work even better if the shelf query points are properly initialized based on the scaling in the URDF. I will remind myself to replace these demos with a new set where the point clouds are obtained from the properly scaled shelves.

RE the gripper point cloud:
This is another mistake I made when collecting the demos which didn't seem to have a negative effect on the performance. Basically, I took the points sampled on the surface of the panda-gripper body and mistakenly transformed them to the position in between the fingers in the grasp pose rather than the actual position of the body. I did realize this eventually but again noticed that the performance was still high. However, again, the performance might be better if these gripper query points are located higher up above the mug near where the gripper body actually was.

---

@zzilch had this fix for the gripper point cloud intersecting the mug, which is correct:

I just find there is a line in the evaluation:
pregrasp_offset_tf = get_ee_offset(ee_pose=pre_grasp_ee_pose)
Is that mean I need to apply this end effector offset to the ee_pose_world of the demo data to get the real point cloud? I tried and it seems more normal.

@kirby516 This is the fix to visualize the gripper points in their correct location -- they just need to be offset by the distance between the body and the fingers. Sorry for this error and the confusion!

@zzilch also had this fix for the shelf intersection, which is also correct:

For shelves, I just need to rescale them according to the scale in the URDF file.
Then everything looks right.