microsoft/VideoX

About the performance of UAV123 in SeqTrack.

Tchuanm opened this issue · 2 comments

Hi, Xin, thanks for your novel work.
With following the same setting with your repo, I evaluated the performance of SeqTrack-B256 on UAV123, as below:

uav | AUC | OP50 | OP75 | Precision | Norm Precision |
seqtrack_b256 | 68.62 | 83.98 | 62.59 | 89.40 | 84.25 |

There is a little difference (lower ~0.5%) compared with evaluated your raw_result fold.
uav | AUC | OP50 | OP75 | Precision | Norm Precision |
seqtrack_b256 | 69.15 | 84.68 | 63.06 | 89.98 | 84.78 |

I have not change anything after clone the repo, do you have some idea about the phenomena.

We found the same matrix multiplication may have slight difference on different devices, leading to the little difference in scores.

This difference is usually invisible on a common task, but the difference accumulates more in sequence modeling (since the prediction is causal), resulting in a slight difference in results.

Thanks for your response. Another interested thing is about the training time. Could you share me around how many hours (or days) of training (i.e., 4 datasets with SeqTrack-B256), which help me to estimate my develop cycle.