Perhaps directly using mask prompt or box prompt in SAM 2 (instead of Uniform Point Sampling)?

[ronghanghu]

Thanks for the great work! Chiming in to provide more details regarding SAM 2 prompting:

Regarding mask prompts in SAM 2:

• SAM 2 also supports using masks as a prompt -- We have an interface add_new_mask for it in the video predictor class SAM2VideoPredictor

Regarding box prompts in SAM 2:

• SAM 2 also supports using bounding box as a prompt -- We just added an example in the video predictor notebook in facebookresearch/sam2#174 to provide a box prompt example.

Maybe we could directly use the box prompt or mask prompt instead of Uniform Point Sampling in this case?

[rentainhe]

Hi ronghang! Thank you so much for releasing such a powerful model and making the code and weights open-source. We overlooked the Mask Prompts during our development process, so we really appreciate your reminder. We've now added the option to use Mask Prompts in our code, and the tracking results look truly impressive.

Regarding box prompts in SAM 2

Very pleased to see that the SAM 2 video predictor now supports box prompts. We will update to the latest version of SAM 2 and make use of this feature, as it will make the entire process more streamlined!

Using uniform sampling was a compromise in our development process and I think it's an option but not a elegant solution for tracking. In the future, we plan to improve by directly using mask or box prompts to make the entire process more elegant and user-friendly~

[ronghanghu]

@rentainhe Sounds good, thanks for the follow-up!

[rentainhe]

@ronghanghu Thank you very much for releasing such an excellent model. We are really looking forward to your future work!