Siamese-Image-Modeling
By Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, Jifeng Dai
This is the official implementation of the CVPR 2023 paper Siamese Image Modeling for Self-Supervised Vision Representation Learning.
🏠 Introduction
SiameseIM is a new form of self-supervised learning that can learn semantic alignment and spatial sensitivity with a single dense loss. We note the following key observations from SiameseIM:
-
Compared with MIM methods, SiameseIM shows that reconstructing another view helps to obtain good semantic alignment.
-
Compared with ID methods, SiameseIM shows that dense supervision can be applied by matching the dense correspondence between two views strictly through their relative positions.
-
SiameseIM is able to surpass both MIM and ID methods over a wide range of tasks. SiameseIM obtains more improvements in few-shot, long-tail and robustness-concerned scenarios.
📈 Main Results
| ImageNet | COCO | ADE20k | LVIS | Robustness | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FT | LIN | 1% FT | AP box | AP mask | mIoU | AP box | AP box rare | AP mask | AP mask rare | IN-A top-1 | IN-R top-1 | IN-Sketch top-1 | IN-C 1-mCE | |
| MoCo-v3 (ID method) | 83.0 | 76.7 | 63.4 | 47.9 | 42.7 | 47.3 | 37.3 | 25.5 | 35.3 | 25.8 | 32.4 | 49.8 | 35.9 | 55.4 |
| MAE (MIM method) | 83.6 | 68.0 | 51.1 | 51.6 | 45.9 | 48.1 | 40.1 | 29.3 | 38.1 | 29.1 | 35.9 | 48.3 | 34.5 | 48.3 |
| SiameseIM | 84.1 | 78.0 | 65.1 | 52.1 | 46.2 | 51.1 | 40.5 | 30.9 | 38.1 | 30.1 | 43.8 | 52.5 | 38.3 | 57.1 |
| Improve w.r.t. MoCo-v3 | +1.1 | +1.3 | +1.7 | +4.2 | +3.5 | +3.8 | +3.2 | +5.4 | +2.8 | +4.3 | +11.4 | +2.7 | +2.4 | +1.7 |
| Improve w.r.t. MAE | +0.5 | +10.0 | +14.0 | +0.5 | +0.3 | +3.0 | +0.4 | +1.6 | +0.0 | +1.0 | +7.9 | +4.2 | +3.8 | +8.8 |
Note:
(1) Compared with MoCo-v3, SiameseIM improves dense prediction tasks (COCO detection, ADE20k segmentation, LVIS detection) significantly;
(2) Compared with MAE, SiameseIM improves long-tail, few-shot, robustness tasks (ImageNet linear evaluation & few-shot classification, ADE20k segmentation, LVIS detection) significantly;
(3) Notably, ADE20k segmentation and LVIS detection both contain long-tail classes, which put forward high requirement for semantic alignment, and detection tasks, which demand good spatial alignment. Thus, SiameseIM can surpass both MoCo-v3 and MAE by a large margin on these tasks.
🛠️ Usage
Preparation
See prepare.md
Model Checkpoint
See checkpoints.md
Pretrain
See pretrain.md
Finetune
See finetune.md
Linear Evaluation
See linear_eval.md
Few-shot Evaluation
See few_shot.md
COCO & LVIS Detection
We use ViTDet for detection tasks, please refer to detectron2.
ADE20k Segmentation
We follow MAE to use UPerNet for segmentation task, please refer to mmsegmentation.
Robustness Evaluation
We evaluate the ImageNet finetuned model on ImageNet-A, ImageNet-R, ImageNet-Sketch and ImageNet-C datasets.
📃 License
This project is released under the CC-BY-NC 4.0 license.
🖊️ Citing SiameseIM
If you find SiameseIM useful in your research, please consider citing:
@inproceedings{tao2023siamese,
title={Siamese image modeling for self-supervised vision representation learning},
author={Tao, Chenxin and Zhu, Xizhou and Su, Weijie and Huang, Gao and Li, Bin and Zhou, Jie and Qiao, Yu and Wang, Xiaogang and Dai, Jifeng},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={2132--2141},
year={2023}
}