DMT

Code release for the CVPR 2023 paper "Discriminative Co-Saliency and Background Mining Transformer for Co-Salient Object Detection".

Abstract

Most previous co-salient object detection works mainly focus on extracting co-salient cues via mining the consistency relations across images while ignore explicit exploration of background regions. In this paper, we propose a Discriminative co-saliency and background Mining Transformer framework (DMT) based on several economical multi-grained correlation modules to explicitly mine both co-saliency and background information and effectively model their discrimination. Specifically, we first propose a region-to-region correlation module for introducing inter-image relations to pixel-wise segmentation features while maintaining computational efficiency. Then, we use two types of pre-defined tokens to mine co-saliency and background information via our proposed contrast-induced pixel-to-token correlation and co-saliency token-to-token correlation modules. We also design a token-guided feature refinement module to enhance the discriminability of the segmentation features under the guidance of the learned tokens. We perform iterative mutual promotion for the segmentation feature extraction and token construction. Experimental results on three benchmark datasets demonstrate the effectiveness of our proposed method.

Result

The prediction results of our dataset can be download from prediction (jjht).

Environment Configuration

Linux with Python ≥ 3.6
PyTorch ≥ 1.7 and torchvision that matches the PyTorch installation. Install them together at pytorch.org to make sure of this. Note, please check PyTorch version matches that is required by Detectron2.
Detectron2: follow Detectron2 installation instructions.
OpenCV is optional but needed by demo and visualization
pip install -r requirements.txt

Data Preparation

Download the dataset from Baidu Driver (cxx2) and unzip them to './dataset'. Then the structure of the './dataset' folder will show as following:

-- dataset
   |-- train_data
   |   |-- | CoCo9k
   |   |-- | DUTS_class
   |   |-- | DUTS_class_syn
   |   |-- |-- | img_png_seamless_cloning_add_naive
   |   |-- |-- | img_png_seamless_cloning_add_naive_reverse_2
   |-- test_data
   |   |-- | CoCA
   |   |-- | CoSal2015
   |   |-- | CoSOD3k

-- dataset
   |-- train_data
   |   |-- | 1
   |   |-- |-- | 2cfa7ea6f.jpg
   |   |-- |-- | 2cfa7ea6f.png
   |   |-- | 2
   |   |-- | 3
   |   |-- | 4
   |-- test_data
   |   |-- | 1
   |   |-- | 2
   |   |-- | 3
   |   |-- | 4

Training model

Download the pretrained VGG model from Baidu Driver(sqd5) and put it into ./checkpoint folder.
Run python train.py.
The trained models with satisfactory performance will be saved in ./checkpoint/CVPR2023_Final_Code

Testing model

Download our trained model from Baidu Driver(c87t) and put it into ./checkpoint/CVPR2023_Final_Code folder.
Run python test.py.
The prediction images will be saved in ./prediction.
Run python ./evaluation/eval_from_imgs.py to evaluate the predicted results on three datasets and the evaluation scores will be written in ./evaluation/result.