A Roadmap for Transfer Learning

Introduction

MIT License

This repo is a collection of AWESOME papers, code related with transfer learning, pre-training and domain adaptation etc. Feel free to star and fork. Feel free to let us know the missing papers (issue or pull request).

This repo is also related with our latest survey, Transferability in Deep Learning

Survey | Library | Website | 中文介绍

Overview

Pre-Training Models

Resources

Survey

  • On the Opportunities and Risks of Fondattion Model [pdf]
  • Pre-Trained Models: Past, Present and Future [pdf]
  • Pre-trained Models for Natural Language Processing: A Survey [pdf]

Paper

  • ViT - An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale [ICLR 2021] [Code]
  • Do Better ImageNet Models Transfer Better? [CVPR 2019]
  • GroupNorm - Group Normalization [ECCV 2018]
  • Transformer - Attention Is All You Need [NIPS 2017]
  • LayerNorm - Layer Normalization [arXiv 21 Jul 2016]
  • ResNet - Deep Residual Learning for Image Recognition [CVPR 2016 Best]
  • BatchNorm - Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift [ICML 2015]

Supervised Pre-Training

Paper

Meta-Learning

Resources

Survey

Paper

Causal Learning

Survey

Paper

Unsupervised Pre-Training

Survey

  • Self-supervised Learning: Generative or Contrastive [TKDE 2021]

Generative Learning

Paper

Contrastive Learning

Resources

  • Lightly: A python library for self-supervised learning on images [Library]

Survey

Paper

Task Adaptation

Paper

  • Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning [ACL 2021 Outstanding]
  • How transferable are features in deep neural networks? [NIPS 2014]
  • DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition [ICML 2014]
  • OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks [arXiv 21 Dec 2013]

Catastrophic Forgetting

Resources

Paper

Negative Transfer

Paper

Parameter Efficiency

Resources

Paper

Data Efficiency

Resources

Survey

  • Generalizing from a Few Examples: A Survey on Few-Shot Learning [10 Apr 2019]
  • Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing [arXiv 28 Jul 2021]

Paper

Domain Adaptation

Resources

Survey

  • A Review of Single-Source Deep Unsupervised Visual Domain Adaptation [1 Sep 2020]
  • Transfer Adaptation Learning: A Decade Survey [12 Mar 2019]
  • A Survey on Transfer Learning [KDE 2010]

Theory

Paper

Statistics Matching

Paper

Domain Adversarial Learning

Paper

Paper for Application

Hypothesis Adversarial Learning

Paper

Domain Translation

Paper

Semi-Supervised Learning

Survey

  • An Overview of Deep Semi-Supervised Learning [pdf]
  • Semi-Supervised Learning [pdf]

Paper

Evaluation

Cross-Task Evaluation

Cross-Domain Evaluation