/MSDN

Official PyTorch Implementation of MSDN (CVPR'22)

Primary LanguagePython

MSDN

This is the total codes of paper "MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning" accepted to CVPR'22. This website includes the following materials for testing and checking our results reported in our paper:

  1. The training codes
  2. The testing codes
  3. The trained model

Requirements

The code implementation of MSDN mainly based on PyTorch. All of our experiments run and test in Python 3.8.8. To install all required dependencies:

$ pip install -r requirements.txt

Training

We trained the model on three popular ZSL benchmarks: CUB, SUN and AWA2 following the data split of xlsa17. Please follow TransZero to prepare datasets and extract visual features.

Training Script

$ python MSDN_cub.py
$ python MSDN_sun.py
$ python MSDN_awa2.py

Note: Please load the corresponding setting when aiming at the CZSL task.

Results

We also upload trained models in test branch. Results of our released models using various evaluation protocols on three datasets, both in the conventional ZSL (CZSL) and generalized ZSL (GZSL) settings.

Dataset Acc(CZSL) U(GZSL) S(GZSL) H(GZSL)
CUB 76.1 68.7 67.5 68.1
SUN 65.8 52.2 34.2 41.3
AWA2 70.1 62.0 74.5 67.7

Note: All of above results are run on a server with an AMD Ryzen 7 5800X CPU and a NVIDIA RTX A6000 GPU. The training codes will be released soon.

Testing

Preparing Dataset and Model

We provide trained models (Google Drive) on three different datasets: CUB, SUN, AWA2 in the CZSL/GZSL setting. You can download model files as well as corresponding datasets, and organize them as follows:

.
├── saved_model
│   ├── CUB_MSDN_CZSL.pth
│   ├── CUB_MSDN_GZSL.pth
│   ├── SUN_MSDN_CZSL.pth
│   ├── SUN_MSDN_GZSL.pth
│   ├── AWA2_MSDN_CZSL.pth
│   └── AWA2_MSDN_GZSL.pth
├── data
│   ├── CUB/
│   ├── SUN/
│   └── AWA2/
└── ···

Testing Script

Runing following commands and testing MSDN on different dataset:

CUB Dataset:

$ python Test_CUB.py

SUN Dataset:

$ python Test_SUN.py

AWA2 Dataset:

$ python Test_AWA2.py

Results

Results of our released models using various evaluation protocols on three datasets, both in the conventional ZSL (CZSL) and generalized ZSL (GZSL) settings.

Dataset Acc(CZSL) U(GZSL) S(GZSL) H(GZSL)
CUB 76.1 68.7 67.5 68.1
SUN 65.8 52.2 34.2 41.3
AWA2 70.1 62.0 74.5 67.7

Note: All of above results are run on a server with an AMD Ryzen 7 5800X CPU and a NVIDIA RTX A6000 GPU. The training codes will be released soon.

Citation

If this work is helpful for you, please cite our paper.

@InProceedings{Chen2022MSDN,
    author    = {Chen, Shiming and Hong, Ziming and Xie, Guo-Sen and Yang, Wenhan and Peng, Qinmu and Wang, Kai and Zhao, Jian and You, Xinge},
    title     = {MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning},
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition ( CVPR )},
    year      = {2022}
}

References

Parts of our codes based on: