Evaluate the model
HanwenCao opened this issue · 7 comments
Hi. I have noticed that you have provided ways to test and train. Would you give me a hint on how to evaluate the whole system? I am very new to MTMC. I would like to compare the results with ground truth so that I can tell the potential causes of failure cases, and the situations where MTMC is hard. Thank you! Any suggestion is appreciated.
Hi Hanwen, IDP, IDR, and IDF1 are all common metrics for evaluation. This is the submission code for AI City, so we did not provide the evaluation code. We may update that part in the future. U can find the evaluation system on https://eval.aicitychallenge.org/aicity2020/submission. The illustration of these metrics is here: https://www.aicitychallenge.org/2020-data-and-evaluation/
Thank you, Kevin. The links you provided are helpful.
I have noticed that you have provided the code for visualization: here
I was wondering if you have the usage of that? Thank you!!
Hi, I ran test.sh on the test set and got "tracklets.txt" and "track3.txt". Is "tracklets.txt" the results of multi-object single-camera tracking?
I was confused by looking at that results. Specifically, I found in the same frame of the same video(camera), there could be different vehicles that are assigned the same carID. For example, in frame 0 of camera042, different cars located at different areas in the image are assigned carID 1. Now I try to pick up some of these lines in "tracklets.txt" and list them below.
`
42 1 0 496 121 67 23 0 0
42 1 0 816 235 95 73 0 0
42 1 0 496 121 67 23 0 0
42 1 0 816 235 95 73 0 0
. . .
`
I don't understand how they could be assigned the same carID by tracking algorithm, given the fact that they are in a single frame and located separately in image space.
Also, I found many duplicated lines in "tracklets.txt" as shown above. Would you know the reason?
Thank you!
Hi, hanwen. I guess that's because we are not sure if the specific vehicle only appears in a single view once. But maybe we should add such a filter (the same vehicle at the same time) to remove these FA samples.
Now close the issue
Thank you!
Hi, hanwen. I guess that's because we are not sure if the specific vehicle only appears in a single view once. But maybe we should add such a filter (the same vehicle at the same time) to remove these FA samples.
tracklets.txt has 424146 lines. But after execute python ./identifier/preprocess/extract_img.py there are only 124661 images cropped.
Is such many duplicated vehicles in a same frame of a same video a normal case?