/MUST

PyTorch code for MUST

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

Masked Unsupervised Self-training for Zero-shot Image Classification

This is the PyTorch code of the MUST paper. The repository supports finetuning a CLIP model on unlabeled images from a target domain.

Requirements

  • pytorch 1.10.0
  • timm 0.4.12
  • tensorboardX
  • ftfy

Dataset Setup

Dataset paths are stored in dataset_catalog.json, which need to be modified to local paths. The imagenet dataset follows the standard folder structure. For other datasets, please refer to the scrips from VISSL to download and prepare. CLIP's labels and prompt templates are stored in classes.json and templates.json.

Training

Run the following code on 16 A100 GPUs:

python -m torch.distributed.run --nproc_per_node=16 train.py --dataset [name_of_dataset] --clip_model ViT-B/16

Results

ViT-B/16:

Method ImageNet SUN397 Food101 GTSRB DTD UCF101
CLIP 68.3 64.4 88.7 43.4 44.7 68.8
MUST 77.7 71.8 92.7 65.5 54.1 81.1

ViT-L/14:

Method ImageNet SUN397 Food101 GTSRB DTD UCF101
CLIP 75.5 67.4 92.9 50.6 55.4 77.0
MUST 82.1 74.6 95.3 68.7 62.6 85.7

Citation

@inproceedings{li2022masked,
      title={Masked Unsupervised Self-training for Label-Free Image Classification}, 
      author={Junnan Li and Silvio Savarese and Steven C. H. Hoi},
      year={2023},
      booktitle={ICLR},
}