Set-of-Marks (SoM)

🍇 [Read our arXiv Paper] 🍎 [Project Page]

🔥 We present Set-of-Mark (SoM), simply overlaying a number of spatial and speakable marks on the images, to unleash the visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.

👉 Comparing standard GPT-4V and its combination with SoM Prompting

📍 Mask proposal generation method

Users can select which granularity of masks to generate, and which mode to use between automatic (top) and interactive (bottom). A higher alpha blending value (0.4) is used for better visualization.

🦄 Interleaved Prompt

SoM enables interleaved prompts which include textual and visual content. The visual content can be represented using the region indices.

🎖️ Mark types used in SoM

🌋 Evaluation tasks examples

Use case

🌷 Grounded Reasoning and Cross-Image Reference

In comparison to GPT-4V without SoM, adding marks enables GPT-4V to ground the reasoning on detailed contents of the image (Left). Clear object cross-image references are observed on the right. 17

🏕️ Problem Solving

Case study on solving CAPTCHA. GPT-4V gives the wrong answer with a wrong number of squares while finding the correct squares with corresponding marks after SoM prompting.

🏔️ Knowledge Sharing

Case study on an image of dish for GPT-4V. GPT-4V does not produce a grounded answer with the original image. Based on SoM prompting, GPT-4V not only speaks out the ingredients but also corresponds them to the regions.

🕌 Personalized Suggestion

SoM-pormpted GPT-4V gives very precise suggestions while the original one fails, even with hallucinated foods, e.g., soft drinks

🌼 Tool Usage Instruction

Likewise, GPT4-V with SoM can help to provide thorough tool usage instruction , teaching users the function of each button on a controller. Note that this image is not fully labeled, while GPT-4V can also provide information about the non-labeled buttons.

🌻 2D Game Planning

GPT-4V with SoM gives a reasonable suggestion on how to achieve a goal in a gaming scenario.

🕌 Simulated Navigation

🌳 Results

We conduct experiments on various vision tasks to verify the effectiveness of our SoM. Results show that GPT4V+SoM outperforms specialists on most vision tasks and is comparable to MaskDINO on COCO panoptic segmentation.

♥️ Acknowledgement

Our model adopts the following models to propose masks:

We also thank GPT-4V for providing a strong foundational model!

✒️ Citation

If you find our work helpful for your research, please consider citing the following BibTeX entry.

@article{yang2023setofmark,
      title={Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V}, 
      author={Jianwei Yang and Hao Zhang and Feng Li and Xueyan Zou and Chunyuan Li and Jianfeng Gao},
      journal={arXiv preprint arXiv:2310.11441},
      year={2023},
}

LinMu7177/SoM