/PVT

Official implementation of PVT series

Primary LanguagePythonApache License 2.0Apache-2.0

Updates

  • (2022/08/09) Application examples for polyp segmentation (polyp-pvt) and vision-language modeling.
  • (2020/06/21) Code of PVTv2 is released! PVTv2 largely improves PVTv1 and works better than Swin Transformer with ImageNet-1K pre-training.

Pyramid Vision Transformer

The image is from Transformers: Revenge of the Fallen.

This repository contains the official implementation of PVTv1 & PVTv2 in image classification, object detection, and semantic segmentation tasks.

Model Zoo

Image Classification

Classification configs & weights see >>>here<<<.

  • PVTv2 on ImageNet-1K
Method Size Acc@1 #Params (M)
PVTv2-B0 224 70.5 3.7
PVTv2-B1 224 78.7 14.0
PVTv2-B2-Linear 224 82.1 22.6
PVTv2-B2 224 82.0 25.4
PVTv2-B3 224 83.1 45.2
PVTv2-B4 224 83.6 62.6
PVTv2-B5 224 83.8 82.0
  • PVTv1 on ImageNet-1K
Method Size Acc@1 #Params (M)
PVT-Tiny 224 75.1 13.2
PVT-Small 224 79.8 24.5
PVT-Medium 224 81.2 44.2
PVT-Large 224 81.7 61.4

Object Detection

Detection configs & weights see >>>here<<<.

  • PVTv2 on COCO

Baseline Detectors

Method Backbone Pretrain Lr schd Aug box AP mask AP
RetinaNet PVTv2-b0 ImageNet-1K 1x No 37.2 -
RetinaNet PVTv2-b1 ImageNet-1K 1x No 41.2 -
RetinaNet PVTv2-b2 ImageNet-1K 1x No 44.6 -
RetinaNet PVTv2-b3 ImageNet-1K 1x No 45.9 -
RetinaNet PVTv2-b4 ImageNet-1K 1x No 46.1 -
RetinaNet PVTv2-b5 ImageNet-1K 1x No 46.2 -
Mask R-CNN PVTv2-b0 ImageNet-1K 1x No 38.2 36.2
Mask R-CNN PVTv2-b1 ImageNet-1K 1x No 41.8 38.8
Mask R-CNN PVTv2-b2 ImageNet-1K 1x No 45.3 41.2
Mask R-CNN PVTv2-b3 ImageNet-1K 1x No 47.0 42.5
Mask R-CNN PVTv2-b4 ImageNet-1K 1x No 47.5 42.7
Mask R-CNN PVTv2-b5 ImageNet-1K 1x No 47.4 42.5

Advanced Detectors

Method Backbone Pretrain Lr schd Aug box AP mask AP
Cascade Mask R-CNN PVTv2-b2-Linear ImageNet-1K 3x Yes 50.9 44.0
Cascade Mask R-CNN PVTv2-b2 ImageNet-1K 3x Yes 51.1 44.4
ATSS PVTv2-b2-Linear ImageNet-1K 3x Yes 48.9 -
ATSS PVTv2-b2 ImageNet-1K 3x Yes 49.9 -
GFL PVTv2-b2-Linear ImageNet-1K 3x Yes 49.2 -
GFL PVTv2-b2 ImageNet-1K 3x Yes 50.2 -
Sparse R-CNN PVTv2-b2-Linear ImageNet-1K 3x Yes 48.9 -
Sparse R-CNN PVTv2-b2 ImageNet-1K 3x Yes 50.1 -
  • PVTv1 on COCO
Detector Backbone Pretrain Lr schd box AP mask AP
RetinaNet PVT-Tiny ImageNet-1K 1x 36.7 -
RetinaNet PVT-Small ImageNet-1K 1x 40.4 -
Mask RCNN PVT-Tiny ImageNet-1K 1x 36.7 35.1
Mask RCNN PVT-Small ImageNet-1K 1x 40.4 37.8
DETR PVT-Small ImageNet-1K 50ep 34.7 -

Semantic Segmentation

Segmentation configs & weights see >>>here<<<.

PVT-v2 + Segmentation see >>>here<<<.

  • PVTv1 on ADE20K
Method Backbone Pretrain Iters mIoU
Semantic FPN PVT-Tiny ImageNet-1K 40K 35.7
Semantic FPN PVT-Small ImageNet-1K 40K 39.8
Semantic FPN PVT-Medium ImageNet-1K 40K 41.6
Semantic FPN PVT-Large ImageNet-1K 40K 42.1

Polyp Segmentation

Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers. pdf | code

Vision-Language Modeling

Masked Vision-Language Transformer in Fashion. pdf | code

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Citation

If you use this code for a paper, please cite:

PVTv1

@inproceedings{wang2021pyramid,
  title={Pyramid vision transformer: A versatile backbone for dense prediction without convolutions},
  author={Wang, Wenhai and Xie, Enze and Li, Xiang and Fan, Deng-Ping and Song, Kaitao and Liang, Ding and Lu, Tong and Luo, Ping and Shao, Ling},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={568--578},
  year={2021}
}

PVTv2

@article{wang2021pvtv2,
  title={Pvtv2: Improved baselines with pyramid vision transformer},
  author={Wang, Wenhai and Xie, Enze and Li, Xiang and Fan, Deng-Ping and Song, Kaitao and Liang, Ding and Lu, Tong and Luo, Ping and Shao, Ling},
  journal={Computational Visual Media},
  volume={8},
  number={3},
  pages={1--10},
  year={2022},
  publisher={Springer}
}

Contact

This repo is currently maintained by Wenhai Wang (@whai362), Enze Xie (@xieenze), and Zhe Chen (@czczup).