SOLIDER: A Python repository from tinyvision

Welcome to SOLIDER! SOLIDER is a Semantic Controllable Self-Supervised Learning Framework to learn general human representations from massive unlabeled human images which can benefit downstream human-centric tasks to the maximum extent. Unlike the existing self-supervised learning methods, prior knowledge from human images is utilized in SOLIDER to build pseudo semantic labels and import more semantic information into the learned representation. Meanwhile, different downstream tasks always require different ratios of semantic information and appearance information, and a single learned representation cannot fit for all requirements. To solve this problem, SOLIDER introduces a conditional network with a semantic controller, which can fit different needs of downstream tasks. For more details, please refer to our paper Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks.

Updates

[2023/07/21: Codes of human pose task is released!]
- Training details of our pretrained model on downstream human pose task is released.
[2023/05/15: Codes of human parsing task is released!]
- Training details of our pretrained model on downstream human parsing task is released.
[2023/04/24: Codes of attribute recognition task is released!]
- Training details of our pretrained model on downstream person attribute recognition task is released.
[2023/03/28: Codes of 3 downstream tasks are released!]
- Training details of our pretrained model on 3 downstream human visual tasks, including person re-identification, person search and pedestrian detection, are released.
[2023/03/13: SOLIDER is accepted by CVPR2023!]
- The paper of SOLIDER is accepted by CVPR2023, and its offical pytorch implementation is released in this repo.

Installation

This codebase has been developed with python version 3.7, PyTorch version 1.7.1, CUDA 10.1 and torchvision 0.8.2.

Datasets

We use LUPerson as our training data, which consists of unlabeled human images. Download LUPerson from its offical link and unzip it.

Training

Choice 1. To train SOLIDER from scratch, please run:

sh run_solider.sh

Choice 2. Training SOLIDER from scratch may take a long time. To speed up the training, you can train a DINO model first, and then finetune it with SOLIDER, as follows:

sh run_dino.sh
sh resume_solider.sh

Finetuning and Inference

There is a demo to run the trained SOLIDER model, which can be embedded into the inference or the downstream task finetuning.

python demo.py

Models

We use Swin-Transformer as our backbone, which shows great advantages on many CV tasks.

Task	Dataset	Swin Tiny (Link)	Swin Small (Link)	Swin Base (Link)
Person Re-identification (mAP/R1) w/o re-ranking	Market1501	91.6/96.1	93.3/96.6	93.9/96.9
	MSMT17	67.4/85.9	76.9/90.8	77.1/90.7
Person Re-identification (mAP/R1) with re-ranking	Market1501	95.3/96.6	95.4/96.4	95.6/96.7
	MSMT17	81.5/89.2	86.5/91.7	86.5/91.7
Attribute Recognition (mA)	PETA_ZS	74.37	76.21	76.43
	RAP_ZS	74.23	75.95	76.42
	PA100K	84.14	86.25	86.37
Person Search (mAP/R1)	CUHK-SYSU	94.9/95.7	95.5/95.8	94.9/95.5
	PRW	56.8/86.8	59.8/86.7	59.7/86.8
Pedestrian Detection (MR-2)	CityPersons	10.3/40.8	10.0/39.2	9.7/39.4
Human Parsing (mIOU)	LIP	57.52	60.21	60.50
Pose Estimation (AP/AR)	COCO	74.4/79.6	76.3/81.3	76.6/81.5

All the models are trained on the whole LUPerson dataset.

Traning codes on Downstream Tasks

Acknowledgement

Our implementation is mainly based on the following codebases. We gratefully thank the authors for their wonderful works.

Reference

If you use SOLIDER in your research, please cite our work by using the following BibTeX entry:

@inproceedings{chen2023beyond,
  title={Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks},
  author={Weihua Chen and Xianzhe Xu and Jian Jia and Hao Luo and Yaohua Wang and Fan Wang and Rong Jin and Xiuyu Sun},
  booktitle={The IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2023},
}

tinyvision/SOLIDER