Validation leaderboard/results

[jspenmar]

Hi,

This benchmark looks really interesting!

I was wondering if you also have a leaderboard for the validation set based on your baselines?
I'm trying to run some of the default configs and would like to check if the results match up.

Kind regards,
Jaime

[eduardohenriquearnold]

Hi,

Thanks for your interest in the dataset & benchmark!

We will provide evaluation results for the paper baselines on the validation set in the next few weeks. In the meantime, if it helps, I can provide validation results for the rotquat_trans baseline below:

  "Average Median Translation Error": 2.3740646709399527,
  "Average Median Rotation Error": 32.38825253157929,
  "Average Median Reprojection Error": 169.1909887180622,
  "Precision @ Pose Error < (25.0cm, 5deg)": 0.03187896798595164,
  "AUC @ Pose Error < (25.0cm, 5deg)": 0.03187896798595164,
  "Precision @ VCRE < 90px": 0.2935296501418344,
  "AUC @ VCRE < 90px": 0.2935296501418344

[jspenmar]

Thanks for the quick reply! Would it be possible to check if the feature matching baselines are roughly correct?

• loftr_pnp_dptkitti

  "Average Median Translation Error": 2.6645722720994938,
  "Average Median Rotation Error": 49.21170162288477,
  "Average Median Reprojection Error": 178.4917427131507,
  "Precision @ Pose Error < (25.0cm, 5deg)": 0.1086046197487505,
  "AUC @ Pose Error < (25.0cm, 5deg)": 0.29989075674421106,
  "Precision @ VCRE < 90px": 0.28366878292584086,
  "AUC @ VCRE < 90px": 0.5686458985731606,
  "Estimates for % of frames": 1.0

• loftr_emat_dptkitti

  "Average Median Translation Error": 2.4845765086396563,
  "Average Median Rotation Error": 40.888409511204586,
  "Average Median Reprojection Error": 181.07176292732873,
  "Precision @ Pose Error < (25.0cm, 5deg)": 0.12913683641766852,
  "AUC @ Pose Error < (25.0cm, 5deg)": 0.30366676812330506,
  "Precision @ VCRE < 90px": 0.3132513845738214,
  "AUC @ VCRE < 90px": 0.5876977039293763,
  "Estimates for % of frames": 1.0

[eduardohenriquearnold]

Hi @jspenmar, apologies for the late reply. These are the results I get for these baselines in the validation set:

• loftr_pnp_dptkitti

{
  "Average Median Translation Error": 2.6645723930765057,
  "Average Median Rotation Error": 49.21170162288477,
  "Average Median Reprojection Error": 178.49174906246384,
  "Precision @ Pose Error < (25.0cm, 5deg)": 0.1086046197487505,
  "AUC @ Pose Error < (25.0cm, 5deg)": 0.29989075674421106,
  "Precision @ VCRE < 90px": 0.28366878292584086,
  "AUC @ VCRE < 90px": 0.5686458985731606,
  "Estimates for % of frames": 1.0
}

• loftr_emat_dptkitti

{
  "Average Median Translation Error": 2.484599186510783,
  "Average Median Rotation Error": 40.82236209471409,
  "Average Median Reprojection Error": 180.96554197447776,
  "Precision @ Pose Error < (25.0cm, 5deg)": 0.12913683641766852,
  "AUC @ Pose Error < (25.0cm, 5deg)": 0.3035588677665686,
  "Precision @ VCRE < 90px": 0.31379170606510876,
  "AUC @ VCRE < 90px": 0.58822019815386,
  "Estimates for % of frames": 1.0
}

They seem consistent with your results.

[eduardohenriquearnold]

Hi @jspenmar , I'll close this issue, feel free to re-open if you have any further queries. Thanks.