uzh-rpg/rpg_trajectory_evaluation

Inconsistency between latex table and boxplot for rel_trans_perc

Closed this issue · 1 comments

Hi Zichao,

I am using the master branch to compare several algorithms in the multi trial mode on the EUROC dataset. In particular, every method runs 5 times on one data mission. I find that the number in "laptop_rel_err....txt" does not agree with the boxplots in overall_rel_trans_perc.pdf.
Below are the numbers in "laptop_rel_err...txt". Notice the translation entry for OKVIS is about 70%.

      &      Translation (\%) &  Rotation (deg/meter)
MSCKF &     0.720 &  0.030 
MSCKF_async &     0.751 &  0.032 
MSCKF_brisk_b2b &     1.001 &  0.028 
MSCKF_klt &     2.030 &  0.047 
OKVIS &     69.306 &  0.074 
OKVIS_nframe &     0.543 &  0.025 

The overall_rel_trans_perc.pdf is attached below. Notice the relative translation errors for OKVIS at different distance intervals are all below 1%.
overall_rel_trans_perc.pdf

I am curious why the translation error for OKVIS reported in "laptop_rel_err...txt" shoots up to 70%?

As I understand, a number reported in "laptop_rel_err...txt" in column Translation is the mean of all rel translation errors for a method, and a middle line in one box of the boxplot in overall_rel_trans_perc.pdf shows the median of all rel translation errors. Is it possible that the mean deviates so much from the median?

When I saved the median instead of the mean for rel_trans_perc, the boxplots agreed with these numbers.

So there is no problem.

It just surprised me that the mean can be so much different from the median even when visually the trajectories matches well to the ground truth.

BTW, I also compared the ATE trans numbers to those obtained by evo with SE(3)(because evo does not support translation + yaw alignment yet), they closely matched. In all, this repo is a great tool.