/self-adaptive-training

[TPAMI2022 & NeurIPS2020] Official implementation of Self-Adaptive Training

Primary LanguagePythonMIT LicenseMIT

Self-Adaptive Training

This is the PyTorch implementation of the

Self-adaptive training significantly improves the generalization of deep networks under noise and enhances the self-supervised representation learning. It also advances the state-of-the-art on learning with noisy label, adversarial training and the linear evaluation on the learned representation.

News

  • 2021.10: The code of Selective Classification for SAT has been released here.
  • 2021.01: We have released the journal version of Self-Adaptive Training, which is a unified algorithm for both the supervised and self-supervised learning. Code for self-supervised learning will be available soon.
  • 2020.09: Our work has been accepted at NeurIPS'2020.

Requirements

  • Python >= 3.6
  • PyTorch >= 1.0
  • CUDA
  • Numpy

Usage

Standard training

The main.py contains training and evaluation functions in standard training setting.

Runnable scripts

  • Training and evaluation using the default parameters

    We provide our training scripts in directory scripts/. For a concrete example, we can use the command as below to train the default model (i.e., ResNet-34) on CIFAR10 dataset with uniform label noise injected (e.g., 40%):

    $ bash scripts/cifar10/run_sat.sh [TRIAL_NAME]

    The argument TRIAL_NAME is optional, it helps us to identify different trials of the same experiments without modifying the training script. The evaluation is automatically performed when training is finished.

  • Additional arguments

    • noise-rate: the percentage of data that being corrupted
    • noise-type: type of random corruptions (i.e., corrupted_label, Gaussian,random_pixel, shuffled_pixel)
    • sat-es: initial epochs of our approach
    • sat-alpha: the momentum term $\alpha$ of our approach
    • arch: the architecture of backbone model, e.g., resnet34/wrn34

Results on CIFAR datasets under uniform label noise

  • Test Accuracy(%) on CIFAR10
Noise Rate 0.2 0.4 0.6 0.8
ResNet-34 94.14 92.64 89.23 78.58
WRN-28-10 94.84 93.23 89.42 80.13
  • Test Accuracy(%) on CIFA100
Noise Rate 0.2 0.4 0.6 0.8
ResNet-34 75.77 71.38 62.69 38.72
WRN-28-10 77.71 72.60 64.87 44.17

Runnable scripts for repreducing double-descent phenomenon

You can use the command as below to train the default model (i.e., ResNet-18) on CIFAR10 dataset with 16.67% uniform label noise injected (i.e., 15% label error rate):

$ bash scripts/cifar10/run_sat_dd_parallel.sh [TRIAL_NAME]
$ bash scripts/cifar10/run_ce_dd_parallel.sh [TRIAL_NAME]

Double-descent ERM vs. single-descent self-adaptive training

Double-descent ERM vs. single-descent self-adaptive training on the error-capacity curve. The vertical dashed line represents the interpolation threshold.

Double-descent ERM vs. single-descent self-adaptive training on the epoch-capacity curve. The dashed vertical line represents the initial epoch E_s of our approach.

Adversarial training

We use state-of-the-art adversarial training algorithm TRADES as our baseline. The main_adv.py contains training and evaluation functions in adversarial training setting on CIFAR10 dataset.

Training scripts

  • Training and evaluation using the default parameters

    We provides our training scripts in directory scripts/cifar10. For a concrete example, we can use the command as below to train the default model (i.e., WRN34-10) on CIFAR10 dataset with PGD-10 attack ($\epsilon$=0.031) to generate adversarial examples:

    $ bash scripts/cifar10/run_trades_sat.sh [TRIAL_NAME]
  • Additional arguments

    • beta: hyper-parameter $1/\lambda$ in TRADES that controls the trade-off between natural accuracy and adversarial robustness
    • sat-es: initial epochs of our approach
    • sat-alpha: the momentum term $\alpha$ of our approach

Robust evaluation script

Evaluate robust WRN-34-10 models on CIFAR10 under PGD-20 attack:

  $ python pgd_attack.py --model-dir "/path/to/checkpoints"

This command evaluates 71-st to 100-th checkpoints in the specified path.

Results

Self-Adaptive Training mitigates the overfitting issue and consistently improves TRADES.

Attack TRADES+SAT

We provide the checkpoint of our best performed model in Google Drive and compare its natural and robust accuracy with TRADES as below.

Attack (submitted by) \ Method TRADES TRADES + SAT
None (initial entry) 84.92 83.48
PGD-20 (initial entry) 56.68 58.03
MultiTargeted-2000 (initial entry) 53.24 53.46
Auto-Attack+ (Francesco Croce) 53.08 53.29

Reference

For technical details, please check the conference version or the journal version of our paper.

@inproceedings{huang2020self,
  title={Self-Adaptive Training: beyond Empirical Risk Minimization},
  author={Huang, Lang and Zhang, Chao and Zhang, Hongyang},
  booktitle={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

@article{huang2021self,
  title={Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning},
  author={Huang, Lang and Zhang, Chao and Zhang, Hongyang},
  journal={arXiv preprint arXiv:2101.08732},
  year={2021}
}

Contact

If you have any question about this code, feel free to open an issue or contact laynehuang@pku.edu.cn.