Inaccurate tracking in case of occlusions

[puranjay123]

@nikitakaraevv

I've noticed that when there's an obstruction in between frames, the model struggles to track points effectively. Is there a possibility of implementing a mechanism to temporarily remove tracking points when an obstruction is detected in the frame, and then resume tracking when the target object reappears in subsequent frames?

I've conducted several tests, and unfortunately, the results haven't been as promising as described in the documentation.

If there's a potential solution or workaround for this issue, I'd greatly appreciate any insights or guidance.

Thank you in advance for your assistance!
This is the Video that I obtained while testing it.
https://github.com/facebookresearch/co-tracker/assets/55429956/c93433fe-adbe-450f-899b-519237eade74

[serycjon]

Hi, can you share the original input video somehow?

[puranjay123]

Hi @serycjon
Original video with a slight trim: You can find the original video with a minor trim at this link: Original Video.

Full 30-second video: If you prefer to access the complete 30-second video, here's the link to it: Full Video.

[nikitakaraevv]

Hi @puranjay123, thanks for the question!

This makes sense because the model was only trained on continuous videos and has never seen merged videos. Therefore, tracking works better if objects are occluded by other objects from the same video, rather than when the model suddenly encounters a completely different video.

For now, the easiest solution is to avoid merging different videos before applying the model.

[puranjay123]

Hi @nikitakaraevv ,
I'll definitely give this a try and avoid using these types of merged videos. If I find a solution, I'll make sure to share it here. Thanks again for the advice!