Not tracking properly
Closed this issue · 7 comments
Hi Rex I'm a big fan and user of your work.
I'm trying to get Cutie to work, but it seems to be failing to track my masks. I have the examples running fine, but when I try to apply this to a video of a drone, it totally fails.
Frame 1 (mask generated externally)
Frame 4 (hasn't moved from initial position)
Frame 37 (track begins to vanish)
I've tried different configs, but no matter what Cutie seems to do this for this particular object. Do you have any idea what be the issue? I can use XMEM but Cutie seems more performant.
This certainly looks unusual (esp. frame 4). Which script are you using?
Similar things would happen if I forgot to sort the input frames (i.e., the script is reading the frames in some sort of random order, and the masks do not align with the input), so that might be happening here.
I'm using scripting_demo.py with some edits to incorporate an object detector. The images are in the correct order. The initial mask is generated from a object detector, so it wouldn't be because the mask files are loaded incorrectly.
I see. Frame 4 looks like that the mask has a roughly correct shape but is offset. This is not a natural failure mode so there might be a bug. Could you please provide the first frame mask and the first four frames (up to frame 4 at least) so that I can investigate? Thanks!
It turns out when I downsample from 4k to 1080, it works perfectly. I should have figured it wasn't trained on such a high resolution.
Glad to see that it works now but that still should not be a failure mode even at very high resolution.
I'm guessing it is something I changed in the config to compensate for OOM errors. I was getting OOM on a H100 80 gb in 2 frames... not sure if that is expected. But it certainly works better on 1080.
Keep it up man your work is such a game changer!