/Awesome-Noisy-Labels

A Survey

Learning from Noisy Labels with Deep Neural Networks: A Survey

This is a repository to help all readers who are interested in handling noisy labels.

If your papers are missing or you have other requests, please contact to ghkswns91@gmail.com.
We will update this repository and paper on a regular basis to maintain up-to-date.

  • Feb 16, 2022: Our survey paper was accepted to TNNLS journal (IF=10.451) [arxiv version]
  • Feb 17, 2022: Last update: including papers published in 2021 and 2022

Citation (.bib) 

@article{song2022survey,
  title={Learning from Noisy Labels with Deep Neural Networks: A Survey},
  author={Song, Hwanjun and Kim, Minseok and Park, Dongmin and Shin, Yooju and Jae-Gil Lee},
  journal={IEEE Transactions on Neural Networks and Learning Systems},
  year={2022}}

Contents

List of Papers with Categorization

All Papers are sorted chronologically according to five categories below, so that you can find related papers more quickly.

We also provide a tabular form of summarization with their methodological comaprison (Table 2 in the paper). - [here]
This is a brief summary for the categorization. Please see Section III in our survey paper for the details - [here]

[Index: Robust Architecture, Robust Regularization, Robust Loss Function, Loss Adjsutment, Sample Selection]

Robust Learning for Noisy Labels
|--- A. Robust Architecture
     |--- A.1. Noise Adaptation Layer: adding a noise adaptation layer at the top of an underlying DNN to learn label transition process
     |--- A.2. Dedicated Architecture: developing a dedicated architecture to reliably support more diverse types of label noises.
|--- B. Robust Regularization
     |--- B.1. Explicit Regularization: an explicit form that modifies the expected tarining loss, e.g., weight decay and dropout.
     |--- B.2. Implicit Regularization: an implicit form that gives the effect of stochasticity, e.g., data augmentation and mini-batch SGD.
|--- C. Robust Loss Function: designing a new loss function robust to label noise.
|--- D. Loss Adjsutment
     |--- D.1. Loss Correction: multiplying the estimated transition matrix to the prediction for all the observable labels.
     |--- D.2. Loss Reweighting: multiplying the estimated example confidence (weight) to the example loss.
     |--- D.3. Label Refurbishment: replacing the original label with other reliable one.
     |--- D.4. Meta Learning: finding an optimal adjustment rule for loss reweighing or label refurbishment.
|--- E. Sample Selection
     |--- E.1. Multi-network Learning: collaborative learning or co-training to identify clean examples from noisy data.
     |--- E.2. Multi-round Learning: refining the selected clean set through training multiple rounds.
     |--- E.3. Hybrid Leanring: combining a specific sample selection strategy with a specific semi-supervised learning model or other orthogonal directions.

In addition, there are some valuable theoretical or empirical papers for understanding the nature of noisy labels.
Go to Theoretical or Empirical Understanding.

A. Robust Architecture

A.1. Noise Adaptation Layer

Year Venue Title Implementation
2015 ICCV Webly supervised learning of convolutional networks Official (Caffe)
2015 ICLRW Training convolutional networks with noisy labels Unofficial (Keras)
2016 ICDM Learning deep networks from noisy labels with dropout regularization Official (MATLAB)
2016 ICASSP Training deep neural-networks based on unreliable labels Unofficial (Chainer)
2017 ICLR Training deep neural-networks using a noise adaptation layer Official (Keras)

A.2. Dedicated Architecture

Year Venue Title Implementation
2015 CVPR Learning from massive noisy labeled data for image classification Official (Caffe)
2018 NeurIPS Masking: A new perspective of noisy supervision Official (TensorFlow)
2018 TIP Deep learning from noisy image labels with quality embedding N/A
2019 ICML Robust inference via generative classifiers for handling noisy labels Official (PyTorch)

B. Robust Regularization

B.1. Explicit Regularization

Year Venue Title Implementation
2018 ECCV Deep bilevel learning Official (TensorFlow)
2019 CVPR Learning from noisy labels by regularized estimation of annotator confusion Official (TensorFlow)
2019 ICML Using pre-training can improve model robustness and uncertainty Official (PyTorch)
2020 ICLR Can gradient clipping mitigate label noise? Unofficial (PyTorch)
2020 ICLR Wasserstein adversarial regularization (WAR) on label noise N/A
2021 ICLR Robust early-learning: Hindering the memorization of noisy labels Official (PyTorch)
2021 ICLR When Optimizing f-Divergence is Robust with Label Noise Official (PyTorch)
2021 ICCV Learning with Noisy Labels via Sparse Regularization Official (PyTorch)
2021 NeurIPS Open-set Label Noise Can Improve Robustness Against Inherent Label Noise Official (PyTorch)

B.2. Implicit Regularization

Year Venue Title Implementation
2015 ICLR Explaining and harnessing adversarial examples Unofficial (PyTorch)
2017 ICLRW Regularizing neural networks by penalizing confident output distributions Unofficial (PyTorch)
2018 ICLR Mixup: Beyond empirical risk minimization Official (PyTorch)
2021 CVPR Augmentation Strategies for Learning with Noisy Labels Official (PyTorch)
2021 CVPR AutoDO: Robust AutoAugment for Biased Data With Label Noise via Scalable Probabilistic Implicit Differentiation Official (PyTorch)

C. Robust Loss Function

Year Venue Title Implementation
2017 AAAI Robust loss functions under label noise for deep neural networks N/A
2017 ICCV Symmetric cross entropy for robust learning with noisy labels Official (Keras)
2018 NeurIPS Generalized cross entropy loss for training deep neural networks with noisy labels Unofficial (PyTorch)
2020 ICLR Curriculum loss: Robust learning and generalization against label corruption N/A
2020 ICML Normalized loss functions for deep learning with noisy labels Official (PyTorch)
2020 ICML Peer loss functions: Learning from noisy labels without knowing noise rates Official (PyTorch)
2021 CVPR Learning Cross-Modal Retrieval with Noisy Labels Official (Pytorch)
2021 CVPR A Second-Order Approach to Learning With Instance-Dependent Label Noise Official (PyTorch)
2022 ICLR An Information Fusion Approach to Learning with Instance-Dependent Label Noise N/A

D. Loss Adjustment

D.1. Loss Correction

Year Venue Title Implementation
2017 CVPR Making deep neural networks robust to label noise: A loss correction approach Official (Keras)
2018 NeurIPS Using trusted data to train deep networks on labels corrupted by severe noise Official (PyTorch)
2019 NeurIPS Are anchor points really indispensable in label-noise learning? Official (PyTorch)
2020 NeurIPS Dual T: Reducing estimation error for transition matrix in label-noise learning N/A
2021 AAAI Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model Official (PyTorch)

D.2. Loss Reweighting

Year Venue Title Implementation
2017 TNNLS Multiclass learning with partially corrupted labels Unofficial (PyTorch)
2017 NeurIPS Active Bias: Training more accurate neural networks by emphasizing high variance samples Unofficial (TensorFlow)

D.3. Label Refurbishment

Year Venue Title Implementation
2015 ICLR Training deep neural networks on noisy labels with bootstrapping Unofficial (Keras)
2018 ICML Dimensionality-driven learning with noisy labels Official (Keras)
2019 ICML Unsupervised label noise modeling and loss correction Official (PyTorch)
2020 NeurIPS Self-adaptive training: beyond empirical risk minimization Official (PyTorch)
2020 ICML Error-bounded correction of noisy labels Official (PyTorch)
2021 AAAI Beyond class-conditional assumption: A primary attempt to combat instancedependent label noise Official (PyTorch)

D.4. Meta Learning

Year Venue Title Implementation
2017 NeurIPSW Learning to learn from weak supervision by full supervision Unofficial (TensorFlow)
2017 ICCV Learning from noisy labels with distillation N/A
2018 ICML Learning to reweight examples for robust deep learning Official (TensorFlow)
2019 NeurIPS Meta-Weight-Net: Learning an explicit mapping for sample weighting Official (PyTorch)
2020 CVPR Distilling effective supervision from severe label noise Official (TensorFlow)
2021 AAAI Meta label correction for noisy label learning Official (PyTorch)
2021 ICCV Adaptive Label Noise Cleaning with Meta-Supervision for Deep Face Recognition N/A

E. Sample Selection

E.1. Multi-network Learning

Year Venue Title Implementation
2017 NeurIPS Decoupling when to update from how to update Official (TensorFlow)
2018 ICML MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels Official (TensorFlow)
2018 NeurIPS Co-teaching: Robust training of deep neural networks with extremely noisy labels Official (PyTorch)
2019 ICML How does disagreement help generalization against label corruption? Official (PyTorch)
2021 CVPR Jo-SRC: A Contrastive Approach for Combating Noisy Labels Official (PyTorch)

E.2. Single- or Multi-round Learning

Year Venue Title Implementation
2018 CVPR Iterative learning with open-set noisy labels Official (Keras)
2019 ICML Learning with bad training data via iterative trimmed loss minimization Official (GluonCV)
2019 ICML Understanding and utilizing deep neural networks trained with noisy labels Official (Keras)
2019 ICCV O2U-Net: A simple noisy label detection approach for deep neural networks Unofficial (PyTorch)
2020 ICMLW How does early stopping can help generalization against label noise? Official (Tensorflow)
2020 NeurIPS A topological filter for learning with label noise Official (PyTorch)
2021 ICLR Learning with Instance-Dependent Label Noise: A Sample Sieve Approach Official (PyTorch)
2021 NeurIPS FINE Samples for Learning with Noisy Labels Official (PyTorch)
2022 ICLR Sample Selection with Uncertainty of Losses for Learning with Noisy Labels N/A

E.3. Hybrid Learning

Year Venue Title Implementation
2019 ICML SELFIE: Refurbishing unclean samples for robust deep learning Official (TensorFlow)
2020 ICLR SELF: Learning to filter noisy labels with self-ensembling N/A
2020 ICLR DivideMix: Learning with noisy labels as semi-supervised learning Official (PyTorch)
2021 ICLR Robust curriculum learning: from clean label detection to noisy label self-correction N/A
2021 NeurIPS Understanding and Improving Early Stopping for Learning with Noisy Labels Official (PyTorch)

Theoretical or Empirical Understanding

How Does a Neural Network’s Architecture Impact Its Robustness to Noisy Labels, NeurIPS 2021 [Link]
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise, AAAI 2021 [Link]
Understanding Instance-Level Label Noise: Disparate Impacts and Treatments, ICML 2021 [Link]
Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations, ICLR 2022 [Link]

Learning from Noisy Labels towards Realistic Scenarios

There have been some studies to solve more realistic setups associated with noisy labels.