PointWSSIS (CVPR 2023)

The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation
Beomyoung Kim^1,2, Joonhyun Jeong^1,2, Dongyoon Han³, Sung Ju Hwang²

¹ _{NAVER Cloud, ImageVision}
² _KAIST
³ _{NAVER AI Lab}

Introduction

We introduce a novel learning scheme named weakly semi-supervised instance segmentation (WSSIS) with point labels for budget-efficient and high-performance instance segmentation. Namely, we consider a dataset setting consisting of a few fully-labeled images and a lot of point-labeled images. Motivated by the main challenge of semi-supervised approaches mainly derives from the trade-off between false-negative and false-positive instance proposals, we propose a method for WSSIS that can effectively leverage the budget-friendly point labels as a powerful weak supervision source to resolve the challenge. Furthermore, to deal with the hard case where the amount of fully-labeled data is extremely limited, we propose a MaskRefineNet that refines noise in rough masks. We conduct extensive experiments on COCO and BDD100K datasets, and the proposed method achieves promising results comparable to those of the fully-supervised model, even with 50% of the fully labeled COCO data (38.8% vs. 39.7%). Moreover, when using as little as 5% of fully labeled COCO data, our method shows significantly superior performance over the state-of-the-art semi-supervised learning method (33.7% vs. 24.9%).

Updates

2023-05-19 First Commit, We release the official implementation of PointWSSIS.

Installation

Our implementation is based on Detectron2 and AdelaiDet.

Please install Detectron2 and AdelaiDet in our repository (there are some modifications, check here).

git clone https://github.com/clovaai/PointWSSIS.git

cd PointWSSIS/detectron2
python3 setup.py build develop --user

cd ../AdelaiDet
python3 setup.py build develop --user

Please check environment requirements and official guideline https://github.com/aim-uofa/AdelaiDet#installation.

Data Preparation

For COCO 2017 dataset, we made subsets 1%, 2%, 5%, 10%, 20%, 30%, 50%.

coco 5p: strong 5% + weak 95%
coco 10p: strong 10% + weak 90%
coco 20p: strong 20% + weak 80%
coco 50p: strong 50% + weak 50%

All subsets are available at google drive

instances_train2017_{subset}_s.json : strong (fully-labeled) data subset.
instances_train2017_{subset}_w.json : weak (point-labeled) data subset.

We also provide our pseudo labels for training the student network:

instances_train2017_{subset}_sw.json : strong and weak (pseudo labels without MaskRefineNet) annotations
instances_train2017_{subset}_sw_refined.json : strong and weak (pseudo labels with MaskRefineNet) annotations

data_root/
    --- coco/
        --- train2017/
        --- val2017/
        --- annotations/
            --- instances_val2017.json
            --- instances_train2017.json
            --- instances_train2017_10p_s.json
            --- instances_train2017_10p_w.json
            --- instances_train2017_10p_sw.json (optional)
            --- instances_train2017_10p_sw_refined.json (optional)

Quick Start

Training teacher network using strong labels
Training MaskRefineNet using strong labels
Generating pseudo labels using weak labels
Training student network using strong and pseudo labels

Run all-in-one script.

bash scripts/coco_5p.sh # coco 5% scenario
bash scripts/coco_10p.sh # coco 10% scenario
bash scripts/coco_20p.sh # coco 20% scenario
bash scripts/coco_50p.sh # coco 50% scenario

Experimental Results

Subset	Teacher Network	MaskRefineNet	Pseudo Labels	Student Network	COCO test-dev
COCO 1%	ckpt	ckpt	json	ckpt	24.0%
COCO 2%	ckpt	ckpt	json	ckpt	25.3%
COCO 5%	ckpt	ckpt	json	ckpt	33.7%
COCO 10%	ckpt	ckpt	json	ckpt	35.8%
COCO 20%	ckpt	ckpt	json	ckpt	37.1%
COCO 30%	ckpt	ckpt	json	ckpt	38.0%
COCO 40%	ckpt	ckpt	json	ckpt	38.3%
COCO 50%	ckpt	ckpt	json	ckpt	38.8%

How to Cite

@inproceedings{kim2023devil,
  title={The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation},
  author={Kim, Beomyoung and Jeong, Joonhyun and Han, Dongyoon and Hwang, Sung Ju},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={11360--11370},
  year={2023}
}

License

PointWSSIS 
Copyright (c) 2023-present NAVER Cloud Corp.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.