/SparK

[ICLR'23 Spotlight] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

Primary LanguagePythonMIT LicenseMIT

SparK: the first successful BERT/MAE-style pretraining on any convolutional networks  Reddit Twitter

This is the official implementation of ICLR paper Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling, which can pretrain any CNN (e.g., ResNet) in a BERT-style self-supervised manner. We've tried our best to make the codebase clean, short, easy to read, state-of-the-art, and only rely on minimal dependencies.

SparK_demo_22s_4k_wo_bages.1.mp4

SOTA  OpenReview  arXiv

🔥 News

🕹️ Colab Visualization Demo

Check pretrain/viz_reconstruction.ipynb for visualizing the reconstruction of SparK pretrained models, like:

We also provide pretrain/viz_spconv.ipynb that shows the "mask pattern vanishing" issue of dense conv layers.

What's new here?

🔥 Pretrained CNN beats pretrained Swin-Transformer:

🔥 After SparK pretraining, smaller models can beat un-pretrained larger models:

🔥 All models can benefit, showing a scaling behavior:

🔥 Generative self-supervised pretraining surpasses contrastive learning:

See our paper for more analysis, discussions, and evaluations.

Todo list

catalog

Pretrained weights (self-supervised; w/o decoder; can be directly finetuned)

Note: for network definitions, we directly use timm.models.ResNet and official ConvNeXt.

reso.: the image resolution; acc@1: ImageNet-1K finetuned acc (top-1)

arch. reso. acc@1 #params flops weights (self-supervised, without SparK's decoder)
ResNet50 224 80.6 26M 4.1G resnet50_1kpretrained_timm_style.pth
ResNet101 224 82.2 45M 7.9G resnet101_1kpretrained_timm_style.pth
ResNet152 224 82.7 60M 11.6G resnet152_1kpretrained_timm_style.pth
ResNet200 224 83.1 65M 15.1G resnet200_1kpretrained_timm_style.pth
ConvNeXt-S 224 84.1 50M 8.7G convnextS_1kpretrained_official_style.pth
ConvNeXt-B 224 84.8 89M 15.4G convnextB_1kpretrained_official_style.pth
ConvNeXt-L 224 85.4 198M 34.4G convnextL_1kpretrained_official_style.pth
ConvNeXt-L 384 86.0 198M 101.0G convnextL_384_1kpretrained_official_style.pth
Pretrained weights (with SparK's UNet-style decoder; can be used to reconstruct images)
arch. reso. acc@1 #params flops weights (self-supervised, with SparK's decoder)
ResNet50 224 80.6 26M 4.1G res50_withdecoder_1kpretrained_spark_style.pth
ResNet101 224 82.2 45M 7.9G res101_withdecoder_1kpretrained_spark_style.pth
ResNet152 224 82.7 60M 11.6G res152_withdecoder_1kpretrained_spark_style.pth
ResNet200 224 83.1 65M 15.1G res200_withdecoder_1kpretrained_spark_style.pth
ConvNeXt-S 224 84.1 50M 8.7G cnxS224_withdecoder_1kpretrained_spark_style.pth
ConvNeXt-L 384 86.0 198M 101.0G cnxL384_withdecoder_1kpretrained_spark_style.pth

Installation & Running

We highly recommended you to use torch==1.10.0, torchvision==0.11.1, and timm==0.5.4 for reproduction. Check INSTALL.md to install all pip dependencies.

Acknowledgement

We referred to these useful codebases:

License

This project is under the MIT license. See LICENSE for more details.

Citation

If you found this project useful, you can kindly give us a star ⭐, or cite us in your work 📖:

@Article{tian2023designing,
  author  = {Keyu Tian and Yi Jiang and Qishuai Diao and Chen Lin and Liwei Wang and Zehuan Yuan},
  title   = {Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling},
  journal = {arXiv:2301.03580},
  year    = {2023},
}