This is an updating survey for Continual Learning of Large Language Models (CL-LLMs), an constantly updated and extended version for the manuscript "Continual Learning of Large Language Models: A Comprehensive Survey".
If you find our survey or this collection of papers useful, please consider citing our work by
@misc{shi2024continual,
title={Continual Learning of Large Language Models: A Comprehensive Survey},
author={Haizhou Shi and Zihao Xu and Hengyi Wang and Weiyi Qin and Wenyuan Wang and Yibin Wang and Hao Wang},
year={2024},
eprint={2404.16789},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (TPAMI 2024) [paper]
- Continual Learning for Large Language Models: A Survey [paper]
- Continual Lifelong Learning in Natural Language Processing: A Survey (COLING 2020) [paper]
- Continual Learning of Natural Language Processing Tasks: A Survey [paper]
- A Survey on Knowledge Distillation of Large Language Models [paper]
- TimeLMs: Diachronic Language Models from Twitter (ACL 2022, Demo Track) [paper][code]
- Continual Pre-Training of Large Language Models: How to (re)warm your model? [paper]
- Continual Learning Under Language Shift [paper]
- Examining Forgetting in Continual Pre-training of Aligned Large Language Models [paper]
- Towards Continual Knowledge Learning of Language Models (ICLR 2022) [paper][code]
- Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (NAACL 2022) [paper]
- TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (EMNLP 2022) [paper][code]
- Continual Training of Language Models for Few-Shot Learning (EMNLP 2022) [paper][code]
- ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding (AAAI 2020) [paper][code]
- Dynamic Language Models for Continuously Evolving Content (KDD 2021) [paper]
- Continual Pre-Training Mitigates Forgetting in Language and Vision [paper][code]
- DEMix Layers: Disentangling Domains for Modular Language Modeling (NAACL 2022) [paper][code]
- Time-Aware Language Models as Temporal Knowledge Bases (TACL 2022) [paper]
- Recyclable Tuning for Continual Pre-training (ACL 2023 Findings) [paper][code]
- Lifelong Language Pretraining with Distribution-Specialized Experts (ICML 2023) [paper]
- ELLE: Efficient Lifelong Pre-training for Emerging Data (ACL 2022 Findings) [paper][code]
- SaulLM-7B: A pioneering Large Language Model for Law [paper][huggingface]
- Lawyer LLaMA Technical Report [paper]
- BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine [paper][code]
- Continuous Training and Fine-tuning for Domain-Specific Language Models in Medical Question Answering [paper]
- PMC-LLaMA: Towards Building Open-source Language Models for Medicine [paper][code]
- AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model [paper]
- Continual Domain-Tuning for Pretrained Language Models [paper]
- HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs [paper][code]
- BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark [paper][code]
- CFGPT: Chinese Financial Assistant with Large Language Model [paper][code]
- Efficient Continual Pre-training for Building Domain Specific Large Language Models [paper]
- WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine [paper][code][huggingface][demo]
- XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters [paper][huggingface]
- AstroLLaMA: Towards Specialized Foundation Models in Astronomy [paper]
- OceanGPT: A Large Language Model for Ocean Science Tasks [paper][code]
- K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilization [paper][code][huggingface]
- MarineGPT: Unlocking Secrets of "Ocean" to the Public [paper][code]
- GeoGalactica: A Scientific Large Language Model in Geoscience [paper][code][huggingface]
- Llemma: An Open Language Model For Mathematics [paper][code][huggingface]
- PLLaMa: An Open-source Large Language Model for Plant Science [paper][code][huggingface]
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis [paper][code][huggingface]
- Code Needs Comments: Enhancing Code LLMs with Comment Augmentation [code]
- StarCoder: may the source be with you! [ppaer][code]
- DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence [paper][code][huggingface]
- IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators [paper][code]
- Code Llama: Open Foundation Models for Code [paper][code]
- LLaMA Pro: Progressive LLaMA with Block Expansion [paper][code][huggingface]
- ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning [paper][code]
- Pre-training Text-to-Text Transformers for Concept-centric Common Sense [paper][code][project]
- Don't Stop Pretraining: Adapt Language Models to Domains and Tasks (ACL 2020) [paper][code]
- EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data [paper]
- Adapting Large Language Models via Reading Comprehension (ICLR 2024) [paper][code]
- Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (NeurIPS 2021) [paper][code]
- Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study (ICLR 2023) [paper][code]
- CIRCLE: Continual Repair across Programming Languages (ISSTA 2022) [paper]
- ConPET: Continual Parameter-Efficient Tuning for Large Language Models [paper][code]
- Enhancing Continual Learning with Global Prototypes: Counteracting Negative Representation Drift [paper]
- Investigating Forgetting in Pre-Trained Representations Through Continual Learning [paper]
- Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models [paper][code]
- LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (ICLR 2022) [paper][code]
- On the Usage of Continual Learning for Out-of-Distribution Generalization in Pre-trained Language Models of Code [paper]
- Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning (ACL 2023 Findings) [paper]
- Parameterizing Context: Unleashing the Power of Parameter-Efficient Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing (NeurIPS 2023) [paper][code]
- Fine-tuned Language Models are Continual Learners [paper][code]
- TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [paper][code]
- Large-scale Lifelong Learning of In-context Instructions and How to Tackle It [paper]
- CITB: A Benchmark for Continual Instruction Tuning [paper][code]
- Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal [paper]
- Don't Half-listen: Capturing Key-part Information in Continual Instruction Tuning [paper]
- ConTinTin: Continual Learning from Task Instructions [paper]
- Orthogonal Subspace Learning for Language Model Continual Learning [paper][code]
- SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models [paper]
- InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions [paper]
- Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors [paper][code]
- On Continual Model Refinement in Out-of-Distribution Data Streams [paper][code][project]
- Melo: Enhancing model editing with neuron-indexed dynamic lora [paper][code]
- Larimar: Large language models with episodic memory control [paper]
- Wilke: Wise-layer knowledge editor for lifelong knowledge editing [paper]
- Alpaca: A Strong, Replicable Instruction-Following Model [project] [code]
- Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems [paper] [code]
- Training language models to follow instructions with human feedback (NeurIPS 2022) [paper]
- Direct preference optimization: Your language model is secretly a reward model (NeurIPS 2023) [paper]
- Copf: Continual learning human preference through optimal policy fitting [paper]
- CPPO: Continual Learning for Reinforcement Learning with Human Feedback (ICLR 2024) [paper]
- A Moral Imperative: The Need for Continual Superalignment of Large Language Models [paper]
- Mitigating the Alignment Tax of RLHF [paper]
- Investigating the Catastrophic Forgetting in Multimodal Large Language Models (PMLR 2024) [paper]
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models [paper] [code]
- Visual Instruction Tuning (NeurIPS 2023, Oral) [paper] [code]
- Continual Instruction Tuning for Large Multimodal Models [paper]
- CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model [paper] [code]
- Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models [paper]
- Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration [paper] [code]