IDEA-Research/Grounded-SAM-2

Real-Time Inference

Henistein opened this issue · 5 comments

Henistein commented

I noticed that in all demo scripts the video is first processed in whole and then propagated, would it be possible to run a video in real-time?

Thank you in advance!

rentainhe commented

Thanks for you issue, I think you can try to use smaller SAM 2 model for faster inference, and maybe you can refer to SAM 2's official repo to see if there're some acceleration methods on this issue

Henistein commented

@rentainhe sorry, maybe I expressed myself badly, what I meant was if would be possible to run grounded sam2 end to end, like in a stream. Imagine I have a real time video stream and I wanted to segment it live while keeping the track of each object, would it be possible?

rentainhe commented

I think SAM 2 now only supports non-stream video input. I have no idea how to support streaming input at this time. Maybe we can refer to the community to see if there are some solutions for this issue.

patrick-tssn commented

@Henistein, I hope my implementation will be helpful, though these is latency.
https://github.com/patrick-tssn/Streaming-Grounded-SAM-2

  • 👍2
  • 🎉2
Masrur02 commented

Hi, @Henistein
Can your code take prompts like the grounded sam2? I mean I want to track road amd cars in a video with continuous id? Is it possible by your implementation??

TIA