l0veKry/nlp_implementation

Implementing nlp papers with PyTorch, Gluon

NLP paper implementation with PyTorch

The papers were implemented in using korean corpus

Classification

• Using the Naver sentiment movie corpus v1.0
• Hyper-parameter was arbitrarily selected. (epoch: 5, mini_batch: 128, except KoBERT (epoch: 2, mini_batch: 32))

	Train ACC (120,000)	Validation ACC (30,000)	Test ACC (50,000)	date
SenCNN	91.98%	86.76%	86.14%	190918
CharCNN	86.54%	82.25%	81.89%	190918
ConvRec	88.94%	83.90%	83.64%	190918
VDCNN	87.05%	84.42%	84.09%	190918
SAN	91.17%	86.54%	86.00%	190919
KoBERT	94.47%	89.85%	89.60%

• Convolutional Neural Networks for Sentence Classification (as SenCNN)
  • https://arxiv.org/abs/1408.5882
• Character-level Convolutional Networks for Text Classification (as CharCNN)
  • https://arxiv.org/abs/1509.01626
• Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers (as ConvRec)
  • https://arxiv.org/abs/1602.00367
• Very Deep Convolutional Networks for Text Classification (as VDCNN)
  • https://arxiv.org/abs/1606.01781
• A Structured Self-attentive Sentence Embedding (as SAN)
  • https://arxiv.org/abs/1703.03130
• BERT_single_sentence_classification (as KoBERT)
  • https://arxiv.org/abs/1810.04805

Paraphrase detection

• Creating dataset from https://github.com/songys/Question_pair
• Hyper-parameter was arbitrarily selected. (epoch: 5, mini_batch: 64)

	Train ACC (6,060)	Validation ACC (1,516)
SAN	91.93%	81.46%
KoBERT	90.80%	93.33%

• A Structured Self-attentive Sentence Embedding (as SAN)
  • https://arxiv.org/abs/1703.03130
• Siamese recurrent architectures for learning sentence similarity
  • https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/viewPaper/12195
• Stochastic Answer Networks for Natural Language Inference
  • https://arxiv.org/abs/1804.07888
• BERT_pairwise_text_classification (as KoBERT)
  • https://arxiv.org/abs/1810.04805

Language model

• Character-Aware Neural Language Models
  • https://arxiv.org/abs/1508.06615

Named entity recognition

• Using the Naver nlp-challange corpus for NER
• Hyper-parameter was arbitrarily selected.

• Bidirectional LSTM-CRF Models for Sequence Tagging
  • https://arxiv.org/abs/1508.01991
• End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
  • https://arxiv.org/abs/1603.01354
• Neural Architectures for Named Entity Recognition
  • https://arxiv.org/abs/1603.01360
• BERT_single_sentence_tagging
  • https://arxiv.org/abs/1810.04805

Neural machine translation

• Effective Approaches to Attention-based Neural Machine Translation
  • https://arxiv.org/abs/1608.07905
• Attention Is All You Need
  • https://arxiv.org/abs/1706.03762

Machine reading comprension

• Machine Comprehension Using Match-LSTM and Answer Pointer
  • https://arxiv.org/abs/1611.01603
• Bi-directional attention flow for machine comprehension
  • https://arxiv.org/abs/1611.01603
• BERT_question_answering
  • https://arxiv.org/abs/1810.04805