Low performance on the nuscenes validation set
Opened this issue · 1 comments
lhiceu commented
Hello @JialianW
I test the performance of the model you provided (nuscenes.pth)on the nuscenes validation set following the readme. But I got low performance which is different from the results in the paper.
### Final results ###
Per-class results:
AMOTA AMOTP RECALL MOTAR GT MOTA MOTP MT ML FAF TP FP FN IDS FRAG TID LGD
bicycle 0.000 1.495 0.552 0.000 1993 0.000 0.793 27 38 683.2 864 35219 893 236 90 1.07 1.82
bus 0.062 1.420 0.224 0.394 2112 0.088 0.811 9 87 17.5 470 285 1638 4 19 0.70 4.02
car 0.088 0.893 0.114 0.520 58317 0.059 0.486 133 3379 55.1 6582 3158 51646 89 166 2.04 4.35
motorcy 0.000 1.579 0.545 0.000 1977 0.000 0.851 29 24 494.5 819 22430 899 259 118 1.14 2.32
pedestr 0.000 1.394 0.639 0.000 25423 0.000 0.873 486 198 2454.6 10887 143963 9168 5368 1778 0.60 1.73
trailer 0.000 1.614 0.543 0.000 2425 0.000 1.084 26 29 2907.8 869 151552 1108 448 169 1.32 2.16
truck 0.003 1.341 0.093 0.139 9650 0.013 0.939 23 509 21.2 884 761 8756 10 30 0.70 3.08
Aggregated results:
AMOTA 0.022
AMOTP 1.391
RECALL 0.387
MOTAR 0.150
GT 14556
MOTA 0.023
MOTP 0.834
MT 733
ML 4264
FAF 947.7
TP 21375
FP 357368
FN 74108
IDS 6414
FRAG 2370
TID 1.08
LGD 2.78
Eval time: 3056.8s
I also tested the results from CenterTrack and got good performance. So I ruled out the issues of data preparation and nuScenes dataset API.
What anything else should I do to get the same performance as you provided? Thank you.
lhiceu commented
Details about opt.txt :
==> commit hash: b'0443c36\n'
==> torch version: 1.3.1
==> cudnn version: 7603
==> Cmd:
['test.py', 'tracking,ddd', '--exp_id', 'nuScenes_3Dtracking', '--dataset', 'nuscenes', '--pre_hm', '--track_thresh', '0.1', '--gpus', '0', '--inference', '--load_model', '../models/nuscenes.pth', '--clip_len', '2', '--trades']
==> Opt:
K: 100
add_05: False
amodel_offset_weight: 1
arch: dla_34
aug_rot: 0
backbone: dla34
batch_size: 32
box_nms: -1
chunk_sizes: [32]
clip_len: 2
custom_dataset_ann_path:
custom_dataset_img_path:
data_dir: /media/he/Disk/TraDeS/src/lib/../../data
dataset: nuscenes
dataset_version:
debug: 0
debug_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking/debug
debugger_theme: white
deform_kernel_size: 3
demo:
dense_reg: 1
dep_weight: 1
depth_scale: 1
dim_weight: 1
dla_node: dcn
down_ratio: 4
efficient_level: 0
embedding: False
eval_val: False
exp_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd
exp_id: nuScenes_3Dtracking
fix_res: True
fix_short: -1
flip: 0.5
flip_test: False
fp_disturb: 0
gpus: [0]
gpus_str: 0
head_conv: {'hm': [256], 'reg': [256], 'wh': [256], 'dep': [256], 'rot': [256], 'dim': [256], 'amodel_offset': [256]}
head_kernel: 3
heads: {'hm': 10, 'reg': 2, 'wh': 2, 'dep': 1, 'rot': 8, 'dim': 3, 'amodel_offset': 2}
hm_disturb: 0
hm_hp_weight: 1
hm_weight: 1
hp_weight: 1
hungarian: False
ignore_loaded_cats: []
inference: True
input_h: 448
input_res: 800
input_w: 800
keep_res: False
kitti_split: 3dop
load_model: ../models/nuscenes.pth
load_results:
lost_disturb: 0
lr: 0.000125
lr_step: [60]
ltrb: False
ltrb_amodal: False
ltrb_amodal_weight: 0.1
ltrb_weight: 0.1
map_argoverse_id: False
master_batch_size: 32
max_age: -1
max_frame_dist: 3
model_output_list: False
msra_outchannel: 256
nID: -1
neck: dlaup
new_thresh: 0.1
nms: False
no_color_aug: False
no_pause: False
no_pre_img: False
no_repeat: True
non_block_test: False
not_cuda_benchmark: False
not_idaup: False
not_max_crop: False
not_prefetch_test: False
not_rand_crop: False
not_set_cuda_env: False
not_show_bbox: False
not_show_number: False
num_classes: 10
num_epochs: 70
num_head_conv: 1
num_iters: -1
num_layers: 101
num_stacks: 1
num_workers: 4
nuscenes_att: False
nuscenes_att_weight: 1
off_weight: 1
optim: adam
out_thresh: 0.1
output_h: 112
output_res: 200
output_w: 200
overlap_thresh: -1
pad: 31
pre_hm: True
pre_img: True
pre_thresh: 0.1
print_iter: 0
prior_bias: -4.6
public_det: False
qualitative: False
reg_loss: l1
reset_hm: False
resize_video: False
resume: False
reuse_hm: False
root_dir: /media/he/Disk/TraDeS/src/lib/../..
rot_weight: 1
rotate: 0
same_aug_pre: False
save_all: False
save_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking
save_framerate: 30
save_img_suffix:
save_imgs: []
save_point: [90]
save_results: False
save_video: False
scale: 0
seed: 317
seg: False
seg_feat_channel: 8
shift: 0
show_track_color: True
skip_first: -1
tango_color: False
task: tracking,ddd
test: False
test_dataset: nuscenes
test_focal_length: -1
test_scales: [1.0]
track_thresh: 0.1
tracking: True
tracking_weight: 1
trades: True
trainval: False
transpose_video: False
use_kpt_center: False
use_loaded_results: False
val_intervals: 10000
velocity: False
velocity_weight: 1
video_h: 512
video_w: 512
vis_gt_bev:
vis_thresh: 0.3
weights: {'hm': 1, 'reg': 1, 'wh': 0.1, 'dep': 1, 'rot': 1, 'dim': 1, 'amodel_offset': 1, 'cost_volume': 1.0}
wh_weight: 0.1
window_size: 7
zero_pre_hm: False
zero_tracking: False