/SASA-pytorch

Unofficial implementation of Stand-Alone Self-Attention in Vision Models (obsolete)

Primary LanguagePythonMIT LicenseMIT

PyTorch Implementation of Stand-Alone Self-Attention in Vision Models

This is NOT an official implementation. Please let me know whether this implementation contains any misreadings of the original paper.

Prerequisites

  • Python +3.6
  • pytorch +1.1.0
  • scipy
  • Pillow
  • torchvision

Benchmark (WIP)

Trained with ImageNet. (WIP: CIFAR-10, CIFAR-100)

Backbone network and parameters are based on the official torchvision ResNet and trainer example.

Trained up to 90 epochs / batch 64 on a single NVIDIA 1080Ti GPU, with SGD optimizer with a learning rate of 0.1 which is linearly warmed up for 10 epochs followed by cosine decay. (according to the SASA paper)