Test set performance
Sungmin-Woo opened this issue · 17 comments
Hi @ZikangZhou
Could you describe the training recipe used for the submission?
I got the test results with your model as below but there seems quite large gap compared to the result you got.
minFDE (K=6): 1.277136966106996
MR (K=6): 0.14637267573551055
minADE (K=6): 0.8215039475137975
As it is hard for me to reach your result by just increasing the training epochs, it will be very helpful if you can provide some other hints.
I am looking forward to your reply!
same question, here is my result:
- minFDE (K=6)": 1.2320899802561445
- "MR (K=6)": 0.1369028575816132
- "minADE (K=6)": 0.7993995795805748
Hi, @XianrenYty @Sungmin-Woo
Could you please provide the corresponding results of the validation set? I am curious about the performance gap between val set and test set for the same model. By the way, are the results you provide come from the repo (the provided ckpt) or re-training? Thanks!
Hi, @XianrenYty @Sungmin-Woo Could you please provide the corresponding results of the validation set? I am curious about the performance gap between val set and test set for the same model. By the way, are the results you provide come from the repo (the provided ckpt) or re-training? Thanks!
I verified similar results on validation set as reported in the paper:
minADE - 0.66, minFDE - 0.97
For the test set, both the provided checkpoint (ckpt) and the re-trained model exhibit similar performance, which is quite inferior.
Hi @Sungmin-Woo ,
I also used the HiVT-128 checkpoint provided in this github. I got very similar results with the test split as you mentioned:
FDE k6: 1.2816903181056214
MR k6: 0.147588395633646
ADE k6: 0.8216046525410116
This is not comparable with the results in the paper and I'm curious what could be the reason.
Have you found out the reason?
Hi @Sungmin-Woo ,
I also used the HiVT-128 checkpoint provided in this github. I got very similar results with the test split as you mentioned:
FDE k6: 1.2816903181056214 MR k6: 0.147588395633646 ADE k6: 0.8216046525410116
This is not comparable with the results in the paper and I'm curious what could be the reason.
Have you found out the reason?
No, I failed to find out the reason and moved on to other project recently.
Please let me know here if you find out.
Hi @Sungmin-Woo ,
I also used the HiVT-128 checkpoint provided in this github. I got very similar results with the test split as you mentioned:
FDE k6: 1.2816903181056214 MR k6: 0.147588395633646 ADE k6: 0.8216046525410116
This is not comparable with the results in the paper and I'm curious what could be the reason.
Have you found out the reason?No, I failed to find out the reason and moved on to other project recently. Please let me know here if you find out.
Would you like to provide your Leaderboard Submission Code？Thanks a lot @MasterIzumi @XianrenYty @Sungmin-Woo @YueYao-bot
Hi @Porphy, you could try this:
import matplotlib.pyplot as plt
import numpy as np
import os
from itertools import permutations, product
from typing import Tuple, List, Dict
import json
from tqdm.auto import tqdm
import torch
from argoverse.map_representation.map_api import ArgoverseMap
import sys
sys.path.append('HiVT')
sys.path.append('HiVT/datasets')
from models.hivt import HiVT
from datasets.argoverse_v1_dataset import process_argoverse, get_lane_features, ArgoverseV1Dataset
from utils import TemporalData
from argoverse.map_representation.map_api import ArgoverseMap
from argoverse.data_loading.argoverse_forecasting_loader import ArgoverseForecastingLoader
checkpoint_path = 'checkpoints/HiVT-64/checkpoints/epoch=63-step=411903.ckpt'
model = HiVT.load_from_checkpoint(checkpoint_path=checkpoint_path, parallel=False)
split = 'test'
dataset = ArgoverseV1Dataset(root='data', split=split, local_radius=model.hparams.local_radius)
output_all_k6 = {}
probs_all = {}
for i, inp in enumerate(tqdm(dataset)):
x = inp.x.numpy()
y = inp.y.numpy()
seq_id = inp.seq_id
positions = inp.positions.numpy()
# the location of the ego vehicle at TIMESTAMP=19
origin = inp.origin.numpy().squeeze()
# the index of the focal agent
agent_index = inp.agent_index
# ego_heading at TIMESTAMP=19
ego_heading = inp.theta.numpy()
ro_angle = inp.rotate_angles[agent_index].numpy()
# Global rotation to align with ego vehicle
rotate_mat = np.array([
[np.cos(ego_heading), -np.sin(ego_heading)],
[np.sin(ego_heading), np.cos(ego_heading)]
])
R = np.array([
[np.cos(ro_angle), -np.sin(ro_angle)],
[np.sin(ro_angle), np.cos(ro_angle)]
])
# we recover the agent trajectory from the inputs, just as a sanity check
offset = positions[agent_index, 19, :]
hist = (np.cumsum(-x[agent_index, 20::-1, :], axis=0)[::-1, :] + offset) @ rotate_mat.T + origin
fut = (y[agent_index, :, :] + offset) @ rotate_mat.T + origin
res, res_pi = model(inp)
agt_res = res[:, agent_index, :, :].detach().cpu().numpy() # [6, num_agents, 30, 2]
probs = torch.softmax(res_pi[agent_index], dim=0)
agt_res_origin = (agt_res[:, :, :2] @ R.T + offset) @ rotate_mat.T + origin
probs_all[seq_id] = probs.detach().cpu().numpy()
output_all_k6[seq_id] = agt_res_origin
from argoverse.evaluation.competition_util import generate_forecasting_h5
output_path = 'competition_files/'
generate_forecasting_h5(output_all_k6, output_path, probabilities= probs_all, filename = 'HiVT64') #this might take awhile
Thank you most sincerely. @YueYao-bot
same question, here is my result:
- minFDE (K=6)": 1.2320899802561445
- "MR (K=6)": 0.1369028575816132
- "minADE (K=6)": 0.7993995795805748
@XianrenYty Would you like to provide your Leaderboard Submission Code？I have tried the code that YueYao-bot provided for me, but the result is much worse than yours. And I'm curious about the reasons.Thanks.
Hi @Porphy , the code I provided is HiVT-64. However, the results we listed are HiVT-128, just in case, if you didn't notice it.
Hi @Porphy , the code I provided is HiVT-64. However, the results we listed are HiVT-128, just in case, if you didn't notice it.
Thanks. I did notice that. But I'm very curious about why his results are significantly better than yours, and I don't think it's because of training instability.
Hi @Porphy, I guess this might be achieved by increasing training epochs or using "train + val" data for training. I am currently trying this out.
Hi @Porphy, I guess this might be achieved by increasing training epochs or using "train + val" data for training. I am currently trying this out.
Thanks a lot. Please let me know the results if you finish trying this strategy.
@Sungmin-Woo , @Porphy , I notice that the model should use "model.eval()" to deactivate all dropout layers before validation. With this, the result improves a lot.
@YueYao-bot I just tried it out and was able to achieve the desired results as XianrenYty provided. Thank you very much.
@YueYao-bot Thank you for sharing it!
@YueYao-bot Hello, is this the test code you tried to write by yourself? Could you share with me how to use the trained model and visualize the predictions while reading the test set? There is no test.py file in the open source of the paper, so does the test process need to be found in Argoverse-API?