Question about the ADE and FDE metrics

Question

Question about the ADE and FDE metrics

Closed this issue 9 months ago · 8 comments

Hi Qingzhao, thank you for releasing the code with us!

After following the steps on the readme file, I tested the normal trajectron_map model on nuScenes dataset with single_frame mode. The ADE and FDE I calculated are 7.131660 and 17.831819, but the results on paper are 2.69 and 5.39. I'm wondering if I miss some important steps to reproduce the result or if I need to modify anything locally?

Thanks!

Answer 1 · 2022-07-26T05:56:47.000Z

FYI

For the first step, I not only downloaded the provided formulated test cases from Google Drive but also followed the steps to generate test cases, because you mentioned that for the trajectron_map model we need map data.

For the second step, I downloaded the pre-trained models from Google Drive and unzipped them into directory test/data.

The contents in trajectron_map_nuscenes/model/original:

After preparing the datasets and models, I run the command:
python test.py --dataset nuscenes --model trajectron_map --mode single_frame

Answer 2 · 2022-08-02T03:05:03.000Z

Hi @juyangbai, the ADE (7.13) and FDE (17.83) is the error in l2 distance, which is in meter^2. The smaller numbers 2.69 and 5.39 are actually squared roots of ADE and FDE (i.e., sqrt(ADE) and sqrt(FDE)), which is in meters. I think the numbers you got are as expected.

Answer 3 · 2022-08-02T03:31:44.000Z

Hi @zqzqz, thank you for your reply! Actually, I rewrote the evaluation function to calculate the error in meters, and modified the test_core function, as the figures showed below. The results using the original evaluation function would reach around one hundred, which should be the I2 distance you mentioned.

Answer 4 · 2022-08-02T20:13:05.000Z

Hi @juyangbai, I double confirmed that I can use the formulated test data to reproduce the ADE/FDE number. Your code looks good to me though. I would guess the problem is the test data.

I have a few suggestions to debug this issue:

You can take a look at the visualized trajectories (which should be generated in the data directory already) to confirm if the results are reasonable.
Do not run data generation (i.e., preprocessing.py in data/nuscenes). I have uploaded the map_name.txt to data/nuscenes to address the map issue. Just use the original map_name.txt instead of generating a new one.
Make sure the version of nuScenes is the same (v1.0).

We can discuss your findings later.

Answer 5 · 2022-08-05T00:25:36.000Z

Hi @zqzqz, thank you for taking the time to debug with me! I tried all of your suggestions those days.

I checked the visualized trajectories, and those figures are not reasonable, which verify the calculation of ADE and FDE should be right. Take the first sample as an example:
I replaced the generated map_name.txt with the one you provided. But the results are still the same: ADE (7.13) and FDE (17.83).
I used the nuScenes Mini Dataset (V1.0) + Map expansion pack (V1.1) following the instruction of Trajecton++. Also, I tried the nuScenes Mini Dataset (V1.0) + the Map expansion pack (V1.0). However, the results are still the same. Are you using the full dataset other than the mini dataset?

I would like to share part of the report.log from the Trajectron++ model. It seems the problem might be the map files.

I also tested the GRIP++ model on the Apolloscape dataset, and could successfully reproduce the ADE and FDE results. So I think the whole environment setup should be right.

Answer 6 · 2022-08-05T02:21:43.000Z

@juyangbai Thank you a lot. I see the problem now.

First, I used the full nuScenes dataset along with the map all in v1.0.

Second, according to your error report, I added a patch in code of Trajectron++ map.py. On line 119, add the following (pardon my ugly code).

        for i in range(centers.shape[0]):
            if centers[i, 0] - context_padding_x < 0:
                centers[i, 0] = context_padding_x
            if centers[i, 0] + context_padding_x > padded_map[i].shape[1]:
                centers[i, 0] = padded_map[i].shape[1] - context_padding_x
            if centers[i, 1] - context_padding_y < 0:
                centers[i, 1] = context_padding_y
            if centers[i, 1] + context_padding_y > padded_map[i].shape[2]:
                centers[i, 1] = padded_map[i].shape[2] - context_padding_y

Basically, when Trajectron++ fetches the map, sometimes it reaches the border of the map thus the map size mismatches. You can try again after adding this patch and see if the ADE/FDE drops. I will try to figure out this issue and update my README.

Answer 7 · 2022-08-12T00:16:37.000Z

Hi @zqzqz, thank you for providing the code for me!

I tried to add this patch in map.py, and used the map V1.0 to test. Unfortunately, the results of ADE and FDE didn't drop.
For the version of the nuScenes dataset, I downloaded the formulated test cases from Google Drive, so I think it shouldn't be a problem. I'm wondering if you are using the same formulated test cases from Google Drive.

Do you have any other suggestions to solve this problem? Thanks!

Answer 8 · 2024-04-16T11:14:20.000Z

Hi Qingzhao, thank you for releasing the code with us!嗨，清照，谢谢你和我们一起发布代码！

After following the steps on the readme file, I tested the normal trajectron_map model on nuScenes dataset with single_frame mode. The ADE and FDE I calculated are 7.131660 and 17.831819, but the results on paper are 2.69 and 5.39. I'm wondering if I miss some important steps to reproduce the result or if I need to modify anything locally?按照自述文件中的步骤操作后，我使用single_frame模式在nuScenes数据集上测试了正常的martron_map模型。计算的ADE和FDE I分别为7.131660和17.831819，而文献上的结果分别为2.69和5.39。我想知道我是否错过了一些重要的步骤来重现结果，或者我是否需要在本地修改任何东西？

Thanks! 谢谢你，谢谢

Hi @zqzqz, thank you for taking the time to debug with me! I tried all of your suggestions those days.您好，感谢您抽出时间与我一起调试！我试过你所有的建议。

I checked the visualized trajectories, and those figures are not reasonable, which verify the calculation of ADE and FDE should be right. Take the first sample as an example:我检查了可视化的轨迹，那些数字是不合理的，这验证了ADE和FDE的计算应该是正确的。以第一个示例为例：

I replaced the generated map_name.txt with the one you provided. But the results are still the same: ADE (7.13) and FDE (17.83).我把生成的 map_name.txt 替换成了你提供的。但结果仍然相同：ADE（7.13）和FDE（17.83）。

I used the nuScenes Mini Dataset (V1.0) + Map expansion pack (V1.1) following the instruction of Trajecton++. Also, I tried the nuScenes Mini Dataset (V1.0) + the Map expansion pack (V1.0). However, the results are still the same. Are you using the full dataset other than the mini dataset?我按照Trajecton++的说明使用了 nuScenes Mini Dataset (V1.0) + Map expansion pack (V1.1) 。我也试了 nuScenes Mini Dataset (V1.0) + Map expansion pack (V1.0) 。然而，结果还是一样。您是否使用完整数据集而不是迷你数据集？

I would like to share part of the report.log from the Trajectron++ model. It seems the problem might be the map files.我想分享Trajectron++模型的report.log的一部分。似乎地图文件有问题。

I also tested the GRIP++ model on the Apolloscape dataset, and could successfully reproduce the ADE and FDE results. So I think the whole environment setup should be right.我还在Apolloscape数据集上测试了GRIP++模型，并成功地重现了ADE和FDE结果。所以我认为整个环境设置应该是正确的。

兄弟，在Apolloscape数据集上测试GRIP++模型，你跑了多久啊。我跑了几个小时没结素