Could you please provide the additional metrics in Table 1?

Question

Could you please provide the additional metrics in Table 1?

Opened this issue a year ago · 10 comments

Thanks for your great work. I am also exploring this topic and want to cite your work. I need the additional metrics in Table 1, including mATE, mASE, mAOE, mAVE and mAAE of your method and the reproduced ones. Could you please provide them?

Answer 1 · 2023-02-16T14:57:46.000Z

Thanks for your attention to our work. I am sorry that we only keep the logs for our own models. Here are our detailed metrics in Table 1:
BEVFormer
pts_bbox_NuScenes/mATE: 0.6931, pts_bbox_NuScenes/mASE: 0.2640, pts_bbox_NuScenes/mAOE: 0.3995, pts_bbox_NuScenes/mAVE: 0.8022, pts_bbox_NuScenes/mAAE: 0.1992, pts_bbox_NuScenes/NDS: 0.4574, pts_bbox_NuScenes/mAP: 0.3863
BEVFormer-T
pts_bbox_NuScenes/mATE: 0.6633, pts_bbox_NuScenes/mASE: 0.2683, pts_bbox_NuScenes/mAOE: 0.3930, pts_bbox_NuScenes/mAVE: 0.3742, pts_bbox_NuScenes/mAAE: 0.1839, pts_bbox_NuScenes/NDS: 0.5153, pts_bbox_NuScenes/mAP: 0.4072

Answer 2 · 2023-02-17T08:21:40.000Z

Thanks a lot~

Answer 3 · 2023-02-17T11:13:44.000Z

One more question~
Are the experiments in Table 1 implemented under the same setting as yours, in terms of backbone, image size, and with or without CBGS?

Answer 4 · 2023-03-02T03:38:57.000Z

Sorry for my late reply. Yes, they are conducted under the same experimental setting.

Answer 5 · 2023-03-04T16:28:33.000Z

Hi, do you have the result about the model efficiency, like FPS and Flops?

Answer 6 · 2023-03-05T05:22:35.000Z

We haven't measured that. We plan to measure it in our journal version paper :)

Answer 7 · 2023-04-25T07:37:15.000Z

Hi, in-line with the above question, could you please provide additional metrics in Table 8, including mATE, mASE, mAOE, mAVE and mAAE for your method (ResNet-101DCN w/ BEVFormer + BEVDistill and BEVFormer-T + BEVDistill) ? Thanks much in advance!

Answer 8 · 2023-04-26T01:49:34.000Z

Hi. The logs are missing due to the recycling of the machines from my company:( I am not sure if I can re-access the file system once the machines are ready. But I will rerun the experiments once the code and machines are ready.

Answer 9 · 2023-04-26T01:55:44.000Z

I just found an experiment record for R101 single frame from my chat history. Here is the results, pts_bbox_NuScenes/NDS: 0.4674, pts_bbox_NuScenes/mAP: 0.3888, pts_bbox_NuScenes/mATE: 0.6858, pts_bbox_NuScenes/mASE: 0.2626, pts_bbox_NuScenes/mAOE: 0.3873, pts_bbox_NuScenes/mAVE: 0.7435, pts_bbox_NuScenes/mAAE: 0.1912.

Answer 10 · 2023-10-13T11:07:00.000Z

Hello,

I have a question regarding Table 1. It appears that the performance of the student model differs from the values reported in the original paper.

According to the information provided in the BEVFormer repository (https://github.com/fundamentalvision/BEVFormer), the reported performance (ResNet-50 backbone) is 35.9 for mAP and 25.7 for NDS. However, in Table 1, the values are listed as 42.3 for mAP and 35.2 for NDS.

Could you kindly explain the source of this discrepancy? I appreciate your assistance in clarifying this matter.

Thank you.