/Contrastive-Learning-with-Non-Semantic-Negatives

Robust Contrastive Learning Using Negative Samples with Diminished Semantics (NeurIPS 2021)

Primary LanguagePythonMIT LicenseMIT

Contrastive Learning with Non-Semantic Negatives

This repository is the official implementation of the NeurIPS 2021 paper Robust Contrastive Learning Using Negative Samples with Diminished Semantics.

tl;dr Contrastive learning utilizes positive pairs which preserve semantic information while perturbing superficial features in the training images. Similarly, we propose to generate negative samples to make the model more robust, where only the superfluous instead of the semantic features are preserved.

News

  • Updated scripts and added new checkpoints based on the hyperparameters identified by IFM on the ImageNet-100 dataset.

Preparation

Install PyTorch and check preprocess/ for ImageNet-100 and ImageNet-Texture preprocessing scripts.

Training

The following code is used to pre-train MoCo-v2 + patch / texture-based NS. The major code is developed with minimal modifications from the official implementation.

python moco-non-sem-neg.py -a resnet50 --lr 0.8 --batch-size 512 --moco-m 0.99 --dist-url 'tcp://localhost:10001' \
  --multiprocessing-distributed --world-size 1 --rank 0 \
  --mlp --moco-t 0.2 --aug-plus --cos --moco-k 16384 \
  --robust patch --num-nonsem 1 --alpha 2 --epochs 200 --patch-ratio 16 72 \
  --ckpt_path ./ckpts/mocov2_mocok16384_bs512_lr0.8_nonsem_16_72_noaug_nn1_alpha2_epoch200  \
  /path/to/imagenet-100/ 

python moco-non-sem-neg.py -a resnet50 --lr 0.8 --batch-size 512 --moco-m 0.99 --dist-url 'tcp://localhost:10001' \
  --multiprocessing-distributed --world-size 1 --rank 0 \
  --mlp --moco-t 0.2 --aug-plus --cos --moco-k 16384 \
  --robust texture --num-nonsem 1 --alpha 2 --epochs 200 \
  --ckpt_path ./ckpts/mocov2_mocok16384_bs512_lr0.8_texture_nn1_alpha2_epoch200 \
  /path/to/imagenet-100-texture/ 
  • Change /path/to/imagenet-100/ with the ImageNet-100 dataset directory.
  • Change --alpha and -moco-k to reproduce results with different configurations.

Linear Evaluation

Run following code is used to reproduce MoCo-v2 + patch-based NS model reported in Table 1.

python main_lincls.py -a resnet50 --lr 10.0 --batch-size 128 --epochs 60 \
  --pretrained ./ckpts/mocov2_mocok16384_bs128_lr0.03_nonsem_16_72_noaug_nn1_alpha2_epoch200/checkpoint_0199.pth.tar \
  --dist-url 'tcp://localhost:10001' --multiprocessing-distributed --world-size 1 --rank 0 \
  --ckpt_path ./ckpts/mocov2_mocok16384_bs128_lr0.03_nonsem_16_72_noaug_nn1_alpha2_epoch200 \
  /path/to/imagenet-100/ 

Pre-trained Models

You can download pretrained models here:

k α ImageNet100 Corruption Sketch Stylized Rendition Checkpoints
MoCo 16384 - 77.88±0.28 43.08±0.27 28.24±0.58 16.20±0.55 32.92±0.12 R1, R2, R3
+Texture 16384 2 77.76±0.17 43.58±0.33 29.11±0.39 16.59±0.17 33.36±0.15 R1, R2, R3
+Patch 16384 2 79.35±0.12 45.13±0.35 31.76±0.88 17.37±0.19 34.78±0.15 R1, R2, R3
+Patch 16384 3 75.58±0.52 44.45±0.15 34.03±0.58 18.60±0.26 36.89±0.11 R1, R2, R3
MoCo 8192 - 77.73±0.38 43.22±0.39 28.45±0.36 16.83±0.12 33.19±0.44 R1, R2, R3
+Patch 8192 2 79.54±0.32 45.48±0.20 33.36±0.45 17.81±0.32 36.31±0.37 R1, R2, R3
MoCo* 65536 - 80.00±0.14 45.15±0.42 30.38±0.30 16.68±0.39 30.38±0.30 R1, R2, R3
+Patch 65536 2 81.18±0.09 46.74±0.32 32.46±0.55 17.63±0.14 36.66±0.18 R1, R2, R3
+Patch 16394 2 81.49±0.11 47.48±0.20 34.20±0.40 17.95±0.41 38.45±0.19 R1, R2, R3

* denotes training with the IFM hyperparameters.

BibTeX

@article{ge2021robust,
  title={Robust Contrastive Learning Using Negative Samples with Diminished Semantics},
  author={Ge, Songwei and Mishra, Shlok and Li, Chun-Liang and Wang, Haohan and Jacobs, David},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  year={2021}
}