Need help evaluating on custom dataset
Closed this issue · 7 comments
Hi!
I'm currently trying to use this great repository to evaluate the performance of my own tracker on a dataset i made myself.
The results from the tracker is in this format (which i believe is correct):
A snippet from my results.txt file is shown below.
2,1,0,227,107,129,-1,-1,-1,-1
2,2,2,0,631,359,-1,-1,-1,-1
2,3,367,211,75,28,-1,-1,-1,-1
3,1,0,226,108,130,-1,-1,-1,-1
3,2,4,0,630,358,-1,-1,-1,-1
3,3,365,211,75,28,-1,-1,-1,-1
As far as i understand the ground truth file is supposed to have the same format (which is has).
However, i'm having troubles with running the run_mot_challenge.py script:
raise TrackEvalException('ini file does not exist: ' + seq + '/' + os.path.basename(ini_file))
trackeval.utils.TrackEvalException: ini file does not exist: ShipSeq-01/seqinfo.ini
It seems like it can't locate the seqinfo.ini file even though it is there. It might be because im very confused about the seqmaps and seqinfo files and how i'm supposed to modify them to my own dataset. Do you have any ideas of what i should do?
Regards
Yes you need to create the .ini files for your gt data in the correct format and put them in the correct file path.
Here is an example I quickly whipped up for a 'custom dataset' based on a previous issue (#25).
The specific issue you have is that you haven't made a seqinfo.ini file under the ShipSeq-01 folder.
E.g. if you look at the example I sent above, it looks like this:
data/gt/dataset_name[ai_city]/split_name[ai_city-train]/sequence_name[ai_city]/seq_info.ini
Once you have a seq_info.ini in this correct location this error will go away.
In terms of editing the seqinfo.ini FOR YOU GT, the MOST IMPORTANT field (the only one that's used actually), is the 'seqLength' e.g. the number of GT frames. This is read from this file because this information is not present alone in the GT file, e.g. there might be a number of frames at the end of sequence with no GT dets, but we still want to allow (and penalise as FPs) predictions in these frames.
I won't close this until you let me know if this worked for you.
Cheers,
Jono
Thanks! I got it to work now!
Yes!
I would suggest to download the examples files which can be found in the README. Then it's a matter of copying the structure of these file. I tried to copy the MOT15 challenge GT and tracker output files.
If your GT and tracker output files are in the MOT15 format they need to be put in the correct folders. I have my ground truth files in their own separate folder for each sequence. For sequence 1 it is in this folder: TrackEval/data/gt/mot_challenge/MyDataset/seq-01/gt
Sequence 2 is in this folder:
TrackEval/data/gt/mot_challenge/MyDataset/seq-02/gt
and so on.
Inside the seq-01 folder is the seqinfo.ini file. Here you need to edit the seqLength to match how many frames are in the specific sequence. This needs to be done for all sequences.
All of the trackers output files should be located in the same folder: TrackEval/data/trackers/mot_challenge/MyDataset/data/data
When the location of the files is correct you need to create the seqmap files in the TrackEval/data/gt/mot_challenge/seqmaps
folder. Do exactly as is done in the examples. I have one that is called MyDataset-train.txt. I have the following in my file:
name
seq-01
seq-02
seq-03
When all of this is done you can run the EvalTracker by typing
python scripts/run_mot_challenge.py --BENCHMARK MyDataset --DO_PRPROC False
That's more or less how i got it to work
@HansEge Thanks for the reply
So the file structure should be
data/
gt/
mot_challenge/
MyDataset/
seq-01/
gt/
gt.txt # <---- ground truth
seqinfo.ini
trackers/
mot_challenge/
MyDataset/
data/
seq-01.txt # <---- model result
like that?
Almost. Seqinfo.ini should be one folder up like below
data/
gt/
mot_challenge/
MyDataset/
seq-01/
seqinfo.ini # <---- seqinfo here
gt/
gt.txt # <---- ground truth
trackers/
mot_challenge/
MyDataset/
data/
seq-01.txt # <---- model result
@HansEge How do you label your ground-truth dataset ? I know how to use the tool to annotate the bounding boxes but don't know how to create ID and link between frames in a video.