/confidentlearning-reproduce

Official data release to reproduce Confident Learning paper results

Primary LanguageJupyter NotebookOtherNOASSERTION

confidentlearning-reproduce

Experimental data for reproducibility of CIFAR-10 experimental results in the confident learning paper.

The code to generate these Confident Learning CIFAR-10 benchmarking results is available in the cleanlab Python package, specifically in examples/cifar10. We used cleanlab v0.1.0 for the original paper.

Because GitHub limits filesizes to 100MB, I cannot upload trained ResNet-50 models (180MB each), but for every setting, I upload an out log file with the accuracy at every batch and test accuracy at every epoch. The file naming conventions are as follows

  • out -- the log files during training
  • train_mask.npy -- boolean vector for which examples where pruned during training
  • cifar10__train__model_resnet50__pyx.npy -- Cross-validation out of sample predicted probabilities for CIFAR-10 under the given noisy labels settings
  • cifar10_noisy_labels -- folder containing all the noisy labels settings
  • experiments.bash -- examples of the commands run to generate results
  • cifar10_train_crossval.py -- training script to perform all cifar-10 experiments (get cross-validated probabilities, evaluate on test set, train on a masked input to remove noisy examples)

A PyTorch-ready version of CIFAR-10 dataset

is available here for download: cifar10/dataset

Need out-of-sample predicted probabilities for CIFAR-10 train set?

You can obtain standard (no noise added to label) predicted probabilities here.

These are computed using four-fold cross-validation with a ResNet50 architecture. You can download the out-of-sample predicted probabilities for all training examples in CIFAR-10 for various noise and sparsities settings here:

  • Noise: 0% | Sparsity: 0% | [LINK]
  • Noise: 20% | Sparsity: 0% | [LINK]
  • Noise: 40% | Sparsity: 0% | [LINK]
  • Noise: 70% | Sparsity: 0% | [LINK]
  • Noise: 20% | Sparsity: 20% | [LINK]
  • Noise: 40% | Sparsity: 20% | [LINK]
  • Noise: 70% | Sparsity: 20% | [LINK]
  • Noise: 20% | Sparsity: 40% | [LINK]
  • Noise: 40% | Sparsity: 40% | [LINK]
  • Noise: 70% | Sparsity: 40% | [LINK]
  • Noise: 20% | Sparsity: 60% | [LINK]
  • Noise: 40% | Sparsity: 60% | [LINK]
  • Noise: 70% | Sparsity: 60% | [LINK]

Precomputed label errors for CIFAR-10 train set

Using the psx predicted probabilities above as input, I used cleanlab, the Python package that implements confident learning, to compute the label errors for every confident learning method in the CL paper, for every noise and sparsity setting. The outputs are boolean numpy arrays. They are ordered in the same order as the examples when loaded using torch.utils.data.dataloader. The PyTorch-prepared CIFAR dataset is available here for download: cifar10/dataset. If you load this dataset in PyTorch, indices will match exactly with the label error masks below.

Column headers are formatted as: <sparsity * 10>_<noise * 10>.

METHOD 0_2 2_2 4_2 6_2 0_4 2_4 4_4 6_4 0_7 2_7 4_7 6_7
C_confusion LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK
formula LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK
CL: PBC LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK
CL: PBNR LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK
CL: C+NR LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK LINK

License

Copyright (c) 2017-2020 Curtis Northcutt. Released under the MIT License. See LICENSE for details.