metterian/nlp-papers

Background

• Auto Regressive
• Auto Encoding

Before Pre-trained

• Word2Vec
  • Efficient Estimation of Word Representations in Vector Space
• Seq2Seq
  • Sequence to Sequence Learning with Neural Networks
• RNN
  • Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

After Pre-trained

• Attention Machanism
  • 고정된 벡터에 정보 압축시 정보 손실 발생
  • RNN: Vanishing Gradient
  • NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE)
• Seq2Seq + Attention
• Transformer
  • Attention Is All You Need
• GPT-1
  • OpenAI GPT-1 - Improving Language Understanding by Generative Pre-Training
• BERT
  • https://arxiv.org/abs/1810.04805
• XLNet
  • XLNet: Generalized Autoregressive Pretraining for Language Understanding
• RoBERTa
  • RoBERTa: A Robustly Optimized BERT Pretraining Approach
• ALBERT
  • ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• T5
  • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• ELECTRA
  • ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
• GPT3
  • Language Models are Few-Shot Learners