/PSALM

[ECCV2024] This is an official implementation for "PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model"

Primary LanguagePythonApache License 2.0Apache-2.0

PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model

Zheng Zhang*, Yeyao Ma*, Enming Zhang*, Xiang Bai

* Equal Contribution

Arxiv Paper

Features

  • A powerful extension of the Large Multi-modal Model for generic (panoptic, instance, semantic) segmentation, referring segmentation and interactivate segmentation.
  • Support joint training across multiple segmentation tasks and visual-language tasks.
  • Demonstrates zero-shot capabilities on unseen task, such as open-vocabulary segmentation, generalizaed referring segmentation, and video object segmentation.

teaser

Updates

  • Release evaluation code
  • Release training code

Installation

See Installation instructions.

Getting Started

See Preparing Datasets for PSALM.

See Getting Started with PSALM.

Model Zoo

  • Download PSALM here.

Citation

If you think this work is useful for your research, please use the following BibTeX entry.

@misc{zhang2024psalm,
      title={PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model}, 
      author={Zheng Zhang and Yeyao Ma and Enming Zhang and Xiang Bai},
      year={2024},
      eprint={2403.14598},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

Thanks for awesome works: Mask2former, Mask2former-Simplify and LLaVA. Code is based on these works.