/InferSent

Primary LanguageJupyter Notebook

InferSent

This is a simple implementation of the paper (Supervised Learning of Universal Sentence Representations from Natural Language Inference Data), including training the Natural Language Inference and build sentence representation encoder.

Four Natural Language Inference model was implemented:

  • Baseline: Averaging word embeddings to obtain sentence representations.
  • UniLSTM: A LSTM was applied on the word embeddings, where the last hidden state is considered as sentence representation.
  • SimpleBiLSTM: The last hidden state of forward and backward layers of BiLSTM are concatenated as the sentence representations.
  • BiLSTM: The max pooling was applied to the concatenation of word-level hidden states from both directions of BiLSTM to retrieve sentence representations.

Installation

  1. Create your own conda environment(python==3.6.13)
  2. Install pytorch and relavant dependencies
  • pytorch 
    conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch
  • tensorboard == 2.5.0
  • torchtext == 0.8.1
  • spacy == 3.0.5 
    conda install -c conda-forge spacy
python -m spacy download en_core_web_sm
  1. Download dataset In this code, the SNLI corpus was used to train Natural Language Inference model. The torchtext was used in the code, so when run the train.py at first time, the SMLI dataset will be downloaded automatically into folder ./dataset/.data
  2. Download GloVe word vector when run the train.py at first time, the GloVe will be downloaded automatically into folder ./dataset/.vector_cache
  3. In order to evaluate the sentence representation, the SentEval was used. please follow the instruction to install and clone it into the folder SentEval.

Usage

Pretrained model

The pretrained models can be download in here. Save them into the folder dataset.

Train

For example, training Baseline model.

python train.py --encoder_flag Baseline --output_dir ./output/Baseline

The --encoder_flag can be Baseline / UniLSTM / SimpleBiLSTM / BiLSTM .

Evaluation (SentEval)

For example, evaluating Baseline sentence encoder:

python eval.py --encoder_flag Baseline --output_dir ./output/Baseline

The --eval_type can be SNLI / SentEval / Both :

  • SNLI : evaluating the Natural Language Inference.
  • SentEval : evaluating sentence encoder using SentEval.
  • Both : evaluating above ( SNLI and SentEval ).

Inferentce (Natural Language Inference -- sentence entailment)

For example, a hypothesis: a woman is making music. and a premise a pregnant lady singing on stage while holding a flag behind her.:

python nli.py --model_dir "./output/Baseline" --hypothesis "a woman is making music." --premise "a pregnant lady singing on stage while holding a flag behind her."

It will generate the relation class ( Nature, contradiction and entailment) between hypothesis and premise.

Extract Sentence Feature

For example, extracting the representation of a woman is making music.

python nli.py --model_dir "./output/Baseline" --sentence "a woman is making music. --feature_dir XXX"

It will save the sentence feature into XXX.