nesl/radar-camera-fusion-depth

Qusetion about the use of nuscenes_train_radar_reprojected

Closed this issue · 4 comments

Zhanfury commented

I've noticed in your code that nuscenes_train_radar_reprojected is obtained through multi-frame fusion. However, I observed that both during the training and testing phases of RadarNet and FusionNet, you are utilizing single-frame radar depth. Could you please clarify the purpose of nuscenes_train_radar_reprojected and when it should be used?

thecyclone commented

Hi Zhanfury,

Thank you for your question.

It was one of the inputs we had initially tried -- the idea was that during training, if we reproject many radar points on the same image, we can show our Radarnet model more data during training and it can learn the mapping between the radar and camera image faster and more efficiently.

However, we empirically found out that this approach was not good as backprojecting radar points onto the image was adding more noise to the input data (as the association between radar and image is already very noisy).

I would encourage you to try it as an input to the radarnet model and see if you get better or worse results.

Zhanfury commented

Thank you very much for your reply.
There are another 2 questions I would like to ask you：

  1. I've noticed that for most samples in the nuscene-mini dataset (which I am currently using to debug my code), the quasi depth maps generated by your provided radarnet pretrained model contain only a small number of non-zero values, with the majority being zero. I am wondering if this is to be expected.
  2. I also observed that in both your paper and code, the threshold for the response is set at 0.5. I am curious about how this threshold was determined. For the situation described in 1), I am wondering if adjusting this threshold could potentially improve the results.
thecyclone commented
  1. Yes, the quasi-dense map or the semi-dense map is just that -- semi dense. Most points will be zero. When you say a small number? how small do you mean? It should at least be in the several thousands range.
  2. We found this to be the best empirically. I think there is a tradeoff between the threshold and the number of points you get. Again. I encourage you to try it out yourself.

I will also encourage you to read the paper and look at the figures. The answer to your first question can easily be inferred from the Figure 1 of our paper.

Zhanfury commented

Thank you for your response, and my apologies for any oversight on my part. After delving deeper into the paper, guided by your insights, I've found the answers I was seeking.

  • 👍1