xxlong0/CNMNet

Question for results

Closed this issue · 13 comments

ymingxie commented

Thank you for releasing the code implementation.

I have a question about the results.

I use your pretrained weights and get pred_depth in 7scenes test set. I evaluate these depth maps using https://github.com/xxlong0/CNMNet/blob/master/utils/metric.py#L448, but I cannot obtain identical results as listed in the paper.

the results I got,
| <1.25 | <1.25^2 | <1.25^3 | abs.rel | sq.rel | rmse | remse log | scale. inv
| 0.481 | 0.799 | 0.934 | 0.283 | 0.214 | 0.563 | 0.338 | 0.326

I want to know more details:

  1. How can I get the translation_gt in https://github.com/xxlong0/CNMNet/blob/master/utils/metric.py#L448
  2. Do you scale the depth map to 640x480 before evaluation
  3. It seems that you only evaluate part of images (https://github.com/xxlong0/CNMNet/blob/master/eval.py#L557)
    ...
xxlong0 commented
ymingxie commented

Thank you for your response.

I estimate the images using:

python eval.py eval_refine with dataset.root_dir=/path/dataset  dataset.image_width=256 dataset.image_height=192 k_size=9 --resume_dir ./checkpoint_idepthscale_3.pt

Then I rescale the estimated depth maps to 640x480 using the nearest interpolation.
I use https://github.com/xxlong0/CNMNet/blob/master/utils/metric.py#L448 to evaluate these estimated depth maps (inverse = False and I ignore the parameter "translation_gt" and doesn't scale the ground truth depth because I have no idea about this parameter)
I still can't obtain the right results.
Do you have any idea?
BTW, do you have any plan to release your depth evaluation script?

Thanks a lot for your time!

xxlong0 commented
flamehaze1115 commented

Thank you for your response.

I estimate the images using:

python eval.py eval_refine with dataset.root_dir=/path/dataset  dataset.image_width=256 dataset.image_height=192 k_size=9 --resume_dir ./checkpoint_idepthscale_3.pt

Then I rescale the estimated depth maps to 640x480 using the nearest interpolation.
I use https://github.com/xxlong0/CNMNet/blob/master/utils/metric.py#L448 to evaluate these estimated depth maps (inverse = False and I ignore the parameter "translation_gt" and doesn't scale the ground truth depth because I have no idea about this parameter)
I still can't obtain the right results.
Do you have any idea?
BTW, do you have any plan to release your depth evaluation script?

Thanks a lot for your time!

have added the evaluation code.

ymingxie commented

@flamehaze1115 Have you pushed it? There is no new commit in this repo.

xxlong0 commented

@flamehaze1115 Have you pushed it? There is no new commit in this repo.

Sorry. I forget to merge them. You can check now.

ymingxie commented

Thanks for your evaluation code!

But I still get the wrong result using:

  1. Estimate the images and get pre_depth:
python eval.py eval_refine with dataset.root_dir=/data/7scenes dataset.image_width=256 dataset.image_height=192 k_size=9 resume_dir=./checkpoint_idepthscale_3.pt
  1. Evaluate pre_depth:
python eval.py cal_metrics with seven_scenes_eva_dir=../evaluations_7_scenes_refine/1

result:
mean_l1_error 1.4408600184182794
a<1.25 0.020107381428476824
a<1.25^2 0.04800932231502878
a<1.25^3 0.08640123737546158
abs.rel 0.7452561264080899
sq.rel 1.1780823899680892
rmse 1.528996372900463
rmse_log 1.5917841874563952
scale.inv 0.37417274269019685

flamehaze1115 commented

Thanks for your evaluation code!

But I still get the wrong result using:

  1. Estimate the images and get pre_depth:
python eval.py eval_refine with dataset.root_dir=/data/7scenes dataset.image_width=256 dataset.image_height=192 k_size=9 resume_dir=./checkpoint_idepthscale_3.pt
  1. Evaluate pre_depth:
python eval.py cal_metrics with seven_scenes_eva_dir=../evaluations_7_scenes_refine/1

result:
mean_l1_error 1.4408600184182794
a<1.25 0.020107381428476824
a<1.25^2 0.04800932231502878
a<1.25^3 0.08640123737546158
abs.rel 0.7452561264080899
sq.rel 1.1780823899680892
rmse 1.528996372900463
rmse_log 1.5917841874563952
scale.inv 0.37417274269019685

I got the reason. My model predicts inverse depth map not depth map. So before evaluation, please do 1./pred_depth to get the correct depth map.

ymingxie commented

After doing pred_depth = 1.0 / pred_depth:

mean_l1_error 1.0437871984977312
a<1.25 0.12759380995062886
a<1.25^2 0.44243236353770776
a<1.25^3 0.7847497696434182
abs.rel 0.7003745292317574
sq.rel 0.8995953632301316
rmse 1.096334456896571
rmse_log 0.5294339892988689
scale.inv 0.17929420544494246

flamehaze1115 commented

After doing pred_depth = 1.0 / pred_depth:

mean_l1_error 1.0437871984977312
a<1.25 0.12759380995062886
a<1.25^2 0.44243236353770776
a<1.25^3 0.7847497696434182
abs.rel 0.7003745292317574
sq.rel 0.8995953632301316
rmse 1.096334456896571
rmse_log 0.5294339892988689
scale.inv 0.17929420544494246

It's weird. I will check the code.

xxlong0 commented

After doing pred_depth = 1.0 / pred_depth:

mean_l1_error 1.0437871984977312
a<1.25 0.12759380995062886
a<1.25^2 0.44243236353770776
a<1.25^3 0.7847497696434182
abs.rel 0.7003745292317574
sq.rel 0.8995953632301316
rmse 1.096334456896571
rmse_log 0.5294339892988689
scale.inv 0.17929420544494246

Hello. I found that I upload the old version. Sorry about that. I will upload new ones tomorrow.

xxlong0 commented

I have uploaded the correct code and pre-trained model. I have checked the new code and it should be right. You can evaluate it now, if any problem, feel free to tell me.

ymingxie commented

Many thanks!