Question about the rectified rgb image and rectified depthmap
Closed this issue · 4 comments
Hi !Thank you very much for sharing this amazing work!
I'm studying the matches between origin rbg image and depthmap, so I tried to overlay the rectified rgb image with rectified depthmap, but I found there were some offsets between them, seems that these two are not aligned correctly, I don't know where the problem is, so could you please tell me the reason? Thank you!
Hello,
Thanks for your comments.
There are some increases in the scared dataset which are discussed in the original paper(https://arxiv.org/pdf/2101.01133.pdf).
There may be some time delay between a video frame and its corresponding camera pose which results in a temporal offset between RGB and depth. This is because depth from the keyframe of each sequence was interpolated based on camera poses.
Additionally, in some cases, the provided calibration is not always very accurate and results in imperfect stereo rectification. This is noticeable in videos from datasets 4 and 5.
Lastly, for some keyframes, the depicted scene may have been shifted during the scanning process which again introduces misalignments.
Having said that, if you are using this code, I would not rule out those shifts being a result of some bug. if you could provide a bit more information about the misalignment you found in the data I may be able you investigate more.
Is the alignment issue present in the unrectified depth-rgb pairs?
Hi, and thank you for response!
I also check the origin depth-rgb pairs and undistorted pairs, the whole dataset 1, keyframe_1 in dataset 2 , whole dataset3 and datatset 6 looks good, the other pairs exist alignment offsets, and as you mentioned the datasets 4 and 5 also have the same issue, each dataset has different kinds of offset In horizontal or vertical direction. I did not calculate the alignment error accurately but just putting them together and the error is obvious.
So when I want to get point-matches between two pairs of rgb image(left camera images only) using corresponding point cloud transformation, I always get the wrong points matches.
The original paper did point out some temporal offset but didn't pull out the solution,so is there any way to align the depth and rgb images? I just want to get each two rgb image(left only) pairs' points-matches with depth transformation. Thank you!
Hi again,
It is really hard to suggest a solution because I haven't spent the time to understand what mainly causes those inaccuracies. The temporal offset may be variable, the hand-eye calibration used to register the camera frames to the robot kinematics may be different across the dataset, etc.
In my opinion, the easiest thing you can do to get poses that are synchronized with the RGB frames is to run some SfM package such us COLMAP (https://colmap.github.io/) or Meshroom (https://meshroom-manual.readthedocs.io/en/latest/).
With COLMAP you may be able to provide the SCARED poses as an initial solution or a least easily rescale your scene to match the original dataset. With the new pose sequence, you may be able to interpolate the reference point cloud again and see if you are getting better pixel correspondences.
Thank you again for your good suggestions! I'll give it a try.