About evaluate.py in ETH and Nuscene datasets?

[Xiejc97]

Hello,

Thanks for your good job! I have some questions about the evaluation code.

1. For pedestrians/evaluate.py(https://github.com/StanfordASL/Trajectron-plus-plus/blob/master/experiments/pedestrians/evaluate.py#L76), the predict() function is:
   predictions = eval_stg.predict(scene, timesteps, ph, num_samples=1, min_history_timesteps=7, min_future_timesteps=12, z_mode=False, gmm_mode=True, full_dist=True) # This will trigger grid sampling
   where min_history_timesteps=7 represents (all historical timesteps 8-curent timestep), min_future_timesteps=12 is the prediction timestep.

2. But for nuScenes/evaluate.py(https://github.com/StanfordASL/Trajectron-plus-plus/blob/master/experiments/nuScenes/evaluate.py#L85), you set the ph is [2,4,6]. Thus I want to ask why you can use the trainned 3s model to predict 1s,2s,3s?
   Can I set the parameter prediction_horizon in nuScenes/models/int_ ee/config.json file to 2, 4 and 6 respectively, and trained respectively, and then use the evaluate.py file similar to pedestrians to evaluate the error of prediction time of 1s, 2s and 3s?
   In addition,the predict() function is:
   predictions = eval_stg.predict(scene, timesteps, ph, num_samples=1, min_future_timesteps=8, z_mode=True, gmm_mode=True, full_dist=False) # This will trigger grid sampling
   I want to ask why the min_history_timesteps not set to 8 (because it is set to 8 in the nuScenes/models/int_ ee/config.json file), but uses the default value 1 of the predict function.

I hope you can help answer my questions, thank you!

[kunhub9]

@Xiejc97Hello. Do you also do trajectory prediction? May I ask you some questions. Could you keep your email if you can?

[BorisIvanovic]

Hi @Xiejc97, sorry for my late reply, I hope I can help with the answers below!

1. Back when we developed Trajectron++, there were no standards with respect to prediction horizons/history context lengths/etc (the nuScenes prediction challenge didn't exist yet), as a result we chose multiple prediction horizons in order to gauge the performance degradation of the model over time (from the same history context).
2. You can set the prediction_horizon parameter to whichever values you would like, if you wish to evaluate the model in that way (e.g., if you particularly wanted to evaluate predicting X seconds into the future from X past seconds of data).
3. As mentioned above, there were no standards with respect to prediction horizons/history context lengths/etc for the nuScenes dataset when we were initially working with it (in contrast to the pedestrian datasets, hence the specification of the min_history_timesteps there), that is why we initially seeked to evaluate our method on as wide a set of agents as possible (those with at least 8 timesteps of future data), rather than additionally limiting it to agents with a certain number of historical context too. Our core reasoning was that, in the real world, you do not want to wait for a certain number of timesteps (to collect a certain number of datapoints) before making predictions 😛 Having said that, feel free to change those parameters to suit your desired evaluation parameters!