A comprehensive list of papers about 'A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning'.
Forgetting refers to the loss or deterioration of previously acquired information or knowledge. While the existing surveys on forgetting have primarily focused on continual learning, forgetting is a prevalent phenomenon observed in various other research domains within deep learning. Forgetting manifests in research fields such as generative models due to generator shifts, and federated learning due to heterogeneous data distributions across clients. Addressing forgetting encompasses several challenges, including balancing the retention of old task knowledge with fast learning of new tasks, managing task interference with conflicting goals, and preventing privacy leakage, etc. Moreover, most existing surveys on continual learning implicitly assume that forgetting is always harmful. In contrast, our survey argues that forgetting is a double-edged sword and can be beneficial and desirable in certain cases, such as privacy-preserving scenarios. By exploring forgetting in a broader context, we aim to present a more nuanced understanding of this phenomenon and highlight its potential advantages. Through this comprehensive survey, we aspire to uncover potential solutions by drawing upon ideas and approaches from various fields that have dealt with forgetting. By examining forgetting beyond its conventional boundaries, in future work, we hope to encourage the development of novel strategies for mitigating, harnessing, or even embracing forgetting in real applications.
If you find our paper or this resource helpful, please consider cite:
@article{Forgetting_Survey_2023,
title={A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning},
author={Zhenyi Wang and Enneng Yang and Li Shen and Heng Huang},
journal={arXiv preprint arXiv:2307.09218},
year={2023}
}
Thanks!
- Harmful Forgetting
- Beneficial Forgetting
Harmful forgetting occurs when we desire the machine learning model to retain previously learned knowledge while adapting to new tasks, domains, or environments. In such cases, it is important to prevent and mitigate knowledge forgetting.
| Problem Setting | Goal | Source of forgetting |
|---|---|---|
| Continual Learning | learn non-stationary data distribution without forgetting previous knowledge | data-distribution shift during training |
| Foundation Model | unsupervised learning on large-scale unlabeled data | data-distribution shift in pre-training, fine-tuning |
| Domain Adaptation | adapt to target domain while maintaining performance on source domain | target domain sequentially shift over time |
| Test-time Adaptation | mitigate the distribution gap between training and testing | adaptation to the test data distribution during testing |
| Meta-Learning | learn adaptable knowledge to new tasks | incrementally meta-learn new classes / task-distribution shift |
| Generative Model | learn a generator to appriximate real data distribution | generator shift/data-distribution shift |
| Reinforcement Learning | maximize accumulate rewards | state, action, reward and state transition dynamics |
| Federated Learning | decentralized training without sharing data | model average; non-i.i.d data; data-distribution shift |
Links:
Forgetting in Continual Learning |
Forgetting in Foundation Models |
Forgetting in Domain Adaptation |
Forgetting in Test-Time Adaptation |
Forgetting in Meta-Learning |
Forgetting in Generative Models |
Forgetting in Reinforcement Learning |
Forgetting in Federated Learning
The goal of continual learning (CL) is to learn on a sequence of tasks without forgetting the knowledge on previous tasks.
Links: Task-aware CL | Task-free CL | Online CL | Semi-supervised CL | Few-shot CL | Unsupervised CL | Theoretical Analysis
Task-aware CL focuses on addressing scenarios where explicit task definitions, such as task IDs or labels, are available during the CL process. Existing methods on task-aware CL have explored five main branches: Memory-based Methods | Architecture-based Methods | Regularization-based Methods | Subspace-based Methods | Bayesian Methods.
Memory-based method keeps a memory buffer that stores the examples/knowledges from previous tasks and replay those examples during learning new tasks.
The architecture-based approach avoids forgetting by reducing parameter sharing between tasks or adding parameters to new tasks.
Regularization-based approaches avoid forgetting by penalizing updates of important parameters or distilling knowledge with previous model as a teacher.
Subspace-based methods perform CL in multiple disjoint subspaces to avoid interference between multiple tasks.
Bayesian methods provide a principled probabilistic framework for addressing Forgetting.
Task-free CL refers to a specific scenario that the learning system does not have access to any explicit task information.
In online CL, the learner is only allowed to process the data for each task once.
The presence of imbalanced data streams in CL (especially online CL) has drawn significant attention, primarily due to its prevalence in real-world application scenarios.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| Online Bias Correction for Task-Free Continual Learning | 2023 | ICLR |
| Information-theoretic Online Memory Selection for Continual Learning | 2022 | ICLR |
| SS-IL: Separated Softmax for Incremental Learning | 2021 | ICCV |
| Online Continual Learning from Imbalanced Data | 2020 | ICML |
| Maintaining Discrimination and Fairness in Class Incremental Learning | 2020 | CVPR |
| Imbalanced Continual Learning with Partitioning Reservoir Sampling | 2020 | ECCV |
| GDumb: A Simple Approach that Questions Our Progress in Continual Learning | 2020 | ECCV |
| Large scale incremental learning | 2019 | CVPR |
| IL2M: Class Incremental Learning With Dual Memory | 2019 | ICCV |
| End-to-end incremental learning | 2018 | ECCV |
Semi-supervised CL is an extension of traditional CL that allows each task to incorporate unlabeled data as well.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| Semi-supervised drifted stream learning with short lookback | 2022 | SIGKDD |
| Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning | 2021 | CVPR |
| Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer | 2021 | IJCNN |
| Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild | 2019 | ICCV |
Few-shot CL refers to the scenario where a model needs to learn new tasks with only a limited number of labeled examples per task while retaining knowledge from previously encountered tasks.
Unsupervised CL (UCL) assumes that only unlabeled data is provided to the CL learner.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| Unsupervised Continual Learning in Streaming Environments | 2023 | TNNLS |
| Representational Continuity for Unsupervised Continual Learning | 2022 | ICLR |
| Probing Representation Forgetting in Supervised and Unsupervised Continual Learning | 2022 | CVPR |
| Unsupervised Continual Learning for Gradually Varying Domains | 2022 | CVPRW |
| Co2L: Contrastive Continual Learning | 2021 | ICCV |
| Unsupervised Progressive Learning and the STAM Architecture | 2021 | IJCAI |
| Continual Unsupervised Representation Learning | 2019 | NeurIPS |
Theory or analysis of continual learning
| Paper Title | Year | Conference/Journal |
|---|---|---|
| The Ideal Continual Learner: An Agent That Never Forgets | 2023 | ICML |
| Continual Learning in Linear Classification on Separable Data | 2023 | ICML |
| Theory on Forgetting and Generalization of Continual Learning | 2023 | ArXiv |
| A Theoretical Study on Solving Continual Learning | 2022 | NeurIPS |
| Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgetting | 2022 | ICLR |
| Continual Learning in the Teacher-Student Setup: Impact of Task Similarity | 2022 | ICML |
| Formalizing the Generalization-Forgetting Trade-off in Continual Learning | 2021 | NeurIPS |
| A PAC-Bayesian Bound for Lifelong Learning | 2014 | ICML |
Foundation models are large machine learning models trained on a vast quantity of data at scale, such that they can be adapted to a wide range of downstream tasks.
Links: Forgetting in Fine-Tuning Foundation Models | Forgetting in One-Epoch Pre-training | CL in Foundation Model
When fine-tuning a foundation model, there is a tendency to forget the pre-trained knowledge, resulting in sub-optimal performance on downstream tasks.
Foundation models often undergo training on a dataset for a single pass. As a result, the earlier examples encountered during pre-training may be overwritten or forgotten by the model more quickly than the later examples.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| Measuring Forgetting of Memorized Training Examples | 2023 | ICLR |
| Quantifying Memorization Across Neural Language Models | 2023 | ICLR |
| Analyzing leakage of personally identifiable information in language models | 2023 | S&P |
| How Well Does Self-Supervised Pre-Training Perform with Streaming Data? | 2022 | ICLR |
| The challenges of continuous self-supervised learning | 2022 | ECCV |
| Continual contrastive learning for image classification | 2022 | ICME |
By leveraging the powerful feature extraction capabilities of foundation models, researchers have been able to explore new avenues for advancing continual learning techniques.
The goal of domain adaptation is to transfer the knowledge from a source domain to a target domain.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| Continual Source-Free Unsupervised Domain Adaptation | 2023 | International Conference on Image Analysis and Processing |
| CoSDA: Continual Source-Free Domain Adaptation | 2023 | Arxiv |
| Lifelong Domain Adaptation via Consolidated Internal Distribution | 2022 | NeurIPS |
| Online Domain Adaptation for Semantic Segmentation in Ever-Changing Conditions | 2022 | ECCV |
| FRIDA -- Generative Feature Replay for Incremental Domain Adaptation | 2022 | CVIU |
| Unsupervised Continual Learning for Gradually Varying Domains | 2022 | CVPRW |
| Continual Adaptation of Visual Representations via Domain Randomization and Meta-learning | 2021 | CVPR |
| Gradient Regularized Contrastive Learning for Continual Domain Adaptation | 2021 | AAAI |
| Learning to Adapt to Evolving Domains | 2020 | NeurIPS |
| AdaGraph: Unifying Predictive and Continuous Domain Adaptation through Graphs | 2019 | CVPR |
| ACE: Adapting to Changing Environments for Semantic Segmentation | 2019 | ICCV |
| Adapting to Continuously Shifting Domains | 2018 | ICLRW |
Test time adaptation (TTA) refers to the process of adapting a pre-trained model on-the-fly to unlabeled test data during inference or testin.
Meta-learning, also known as learning to learn, focuses on developing algorithms and models that can learn from previous learning experiences to improve their ability to learn new tasks or adapt to new domains more efficiently and effectively.
Links: Incremental Few-Shot Learning | Continual Meta-Learning
Incremental few-shot learning (IFSL) focuses on the challenge of learning new categories with limited labeled data while retaining knowledge about previously learned categories.
The goal of continual meta-learning (CML) is to address the challenge of forgetting in non-stationary task distributions.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| Adaptive Compositional Continual Meta-Learning | 2023 | ICML |
| Learning to Learn and Remember Super Long Multi-Domain Task Sequence | 2022 | CVPR |
| Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions | 2022 | ECCV |
| Variational Continual Bayesian Meta-Learning | 2021 | NeurIPS |
| Meta Learning on a Sequence of Imbalanced Domains with Difficulty Awareness | 2021 | ICCV |
| Addressing Catastrophic Forgetting in Few-Shot Problems | 2020 | ICML |
| Continuous meta-learning without tasks | 2020 | NeurIPS |
| Reconciling meta-learning and continual learning with online mixtures of tasks | 2019 | NeurIPS |
| Fast Context Adaptation via Meta-Learning | 2019 | ICML |
| Online meta-learning | 2019 | ICML |
The goal of a generative model is to learn a generator that can generate samples from a target distribution.
Links: GAN Training is a Continual Learning Problem | Lifelong Learning of Generative Models
Treating GAN training as a continual learning problem.
The goal is to develop generative models that can continually generate high-quality samples for both new and previously encountered tasks.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| The Curse of Recursion: Training on Generated Data Makes Models Forget | 2023 | Arxiv |
| Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models | 2023 | Arxiv |
| Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models | 2023 | Arxiv |
| Lifelong Generative Modelling Using Dynamic Expansion Graph Model | 2022 | AAAI |
| Continual Variational Autoencoder Learning via Online Cooperative Memorization | 2022 | ECCV |
| Hyper-LifelongGAN: Scalable Lifelong Learning for Image Conditioned Generation | 2021 | CVPR |
| Lifelong Twin Generative Adversarial Networks | 2021 | ICIP |
| Lifelong Mixture of Variational Autoencoders | 2021 | TNNLS |
| Lifelong Generative Modeling | 2020 | Neurocomputing |
| Lifelong GAN: Continual Learning for Conditional Image Generation | 2019 | ICCV |
Reinforcement learning is a machine learning technique that allows an agent to learn how to behave in an environment by trial and error, through rewards and punishments.
Federated learning (FL) is a decentralized machine learning approach where the training process takes place on local devices or edge servers instead of a centralized server.
Links: Forgetting Due to Non-IID Data in FL | Federated Continual Learning
This branch pertains to the forgetting problem caused by the inherent non-IID (not identically and independently distributed) data among different clients participating in FL.
This branch addresses the issue of continual learning within each individual client in the federated learning process, which results in forgetting at the overall FL level.
Beneficial forgetting arises when the model contains private information that could lead to privacy breaches or when irrelevant information hinders the learning of new tasks. In these situations, forgetting becomes desirable as it helps protect privacy and facilitate efficient learning by discarding unnecessary information.
| Problem Setting | Goal |
|---|---|
| Mitigate Overfitting | mitigate memorization of training data through selective forgetting |
| Debias and Forget Irrelevant Information | forget biased information to achieve better performance or remove irrelevant information to learn new tasks |
| Machine Unlearning | forget some specified training data to protect user privacy |
Links: Combat Overfitting Through Forgetting | Learning New Knowledge Through Forgetting Previous Knowledge | Machine Unlearning
Overfitting in neural networks occurs when the model excessively memorizes the training data, leading to poor generalization. To address overfitting, it is necessary to selectively forget irrelevant or noisy information.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier | 2023 | ICLR |
| The Primacy Bias in Deep Reinforcement Learning | 2022 | ICML |
| Learning with Selective Forgetting | 2021 | IJCAI |
| SIGUA: Forgetting May Make Learning with Noisy Labels More Robust | 2020 | ICML |
| Invariant Representations through Adversarial Forgetting | 2020 | AAAI |
| Forget a Bit to Learn Better: Soft Forgetting for CTC-based Automatic Speech Recognition | 2019 | Interspeech |
"Learning to forget" suggests that not all previously acquired prior knowledge is helpful for learning new tasks.
| Paper Title | Year | Conference/Journal |
|---|---|---|
| ReFactor GNNs: Revisiting Factorisation-based Models from a Message-Passing Perspective | 2022 | NeurIPS |
| Fortuitous Forgetting in Connectionist Networks | 2022 | ICLR |
| Skin Deep Unlearning: Artefact and Instrument Debiasing in the Context of Melanoma Classification | 2022 | ICML |
| Near-Optimal Task Selection for Meta-Learning with Mutual Information and Online Variational Bayesian Unlearning | 2022 | AISTATS |
| AFEC: Active Forgetting of Negative Transfer in Continual Learning | 2021 | NeurIPS |
| Active Forgetting: Adaptation of Memory by Prefrontal Control | 2021 | Annual Review of Psychology |
| Learning to Forget for Meta-Learning | 2020 | CVPR |
| The Forgotten Part of Memory | 2019 | Nature |
| Learning Not to Learn: Training Deep Neural Networks with Biased Data | 2019 | CVPR |
| Inhibiting your native language: the role of retrieval-induced forgetting during second-language acquisition | 2007 | Psychological Science |
Machine unlearning, a recent area of research, addresses the need to forget previously learned training data in order to protect user data privacy.
Contact
We welcome all researchers to contribute to this repository 'forgetting in deep learning'.
Email: wangzhenyineu@gmail.com | ennengyang@stumail.neu.edu.cn