Questions about submission results on Waymo open dataset

[ggosjw]

Hi, thank you for open-sourcing your great work.

I have questions regarding my reproduced DIPP submission results on Waymo dataset. My test dataset results are

And my validation results are

Both are worse than your results.
I preprocesssed the data using your data_process.py, changing the history timestep as 11, and future timestep as 80. I find ignoring the ego vehicle imitation loss will slightly improve the prediciton result.
Could you give some intutive suggestions regarding how to further improve the prediction performance.
Much appreciation for any help you can offer.

[MCZhi]

Hi, @ggosjw, thanks for your questions. We have made some modifications to the network and training to achieve a good result on the benchmark. I have some suggestions here that you might find helpful.

1. Remove the differentiable motion planner because here we only focus on the open-loop prediction task. It would result in better FDE and ADE.
2. Focus on the performance of the ego agent. The prediction of other agents can be used as an auxiliary task or just treat other agents as environment information in encoding.
3. Train three separate models for different object types.
4. By doing the above, you are supposed to get good FDE and ADE, but to get a good mAP, you may need to use the ensemble technique such as the EM method proposed in MultiPath++.

[ggosjw]

Got it! Thank you very much!
For the second suggestion, does it mean that I should do ego vehicle prediction instead of imitation learning-based planning?

So the final loss may look like L=a1 * ego_prediction+a2 * surrounding_prediction+a3*score_loss.

Thank you again :)

[MCZhi]

Yes. And please remember to set the weight a2 smaller.