How about 3D version?

Question

How about 3D version?

Opened this issue 5 months ago · 3 comments

I wonder can we tracking object by Grounded Sam2 in 3D?

Answer 1 · 2024-08-19T07:37:27.000Z

Grounded SAM 2 can currently only handle 2D images. If it needs to be applied in a 3D scene, I think you first need to project a certain view onto a 2D plane and then apply Grounded SAM 2.

BTW, I was wondering what's the meaning of tracking objects in 3D scenarios, could you explain the scenario you need to use more clearly? This will help us better assist you in brainstorming solutions.

Answer 2 · 2024-08-19T07:44:07.000Z

I have seen a similar example in SEEM: https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once?tab=readme-ov-file#tulip-nerf-examples, is this the scenarios you need or not

Answer 3 · 2024-08-20T05:46:08.000Z

Grounded SAM 2 can currently only handle 2D images. If it needs to be applied in a 3D scene, I think you first need to project a certain view onto a 2D plane and then apply Grounded SAM 2.

BTW, I was wondering what's the meaning of tracking objects in 3D scenarios, could you explain the scenario you need to use more clearly? This will help us better assist you in brainstorming solutions.

For example, tracking 3D object could be more precise compare to 2D, and tracking multi object is very common in Street scenarios(self-driving) or in some dynamic slam/indoor robot field.