Common-Sense-QA

This repository contains code for training models on different question answering tasks along with code for generating commonsense inferences to use for such tasks.

Introduction

The goal of this project was to assess the extent to which commonsense reasoning plays a role in the quality of answers generated by Question Answering (QA) models.

Directory Structure

.
+- Common-Sense-QA
   +- comet
   |  +- src
   |  |  +- main.py
   |  +- config
   |  |  +- atomic
   |  |     +- changes.json
   |  +- scripts
   |  |  +- data
   |  |  |  +- make_atomic_data_loader.py
   |  |  +-setup
   |  |     +- get_atomic_data.sh
   |  |     +- get_conceptnet_data.sh
   |  |     +- get_model_files.sh
   |  +- load_data.py
   |  +- parameters_names.json
   |  +- run_sqa_cs.py
   |  +- run_squad_cs.py
   |  +- sqa_loader_cs.py
   +- cove
   +- decaNLP
   +- plots
   +- socialiqa
   |  +- run_sqa.py
   |  +- sqa_loader.py
   +- requirements.txt
   +- custom_bert.py
   +- mlt_batch_scheduler.py
   +- multi_task_batch_scheduler.py
   +- run_glue.py
   +- run_squad.py
   +- run_squad_sst.py
   +- torchsampler.py

Installation

pip install -r requirements.txt

Dataset Downloads

SQuAD data can be downloaded from the links given below and should be saved in $SQUAD_DIR directory:
train-v1.1.json
dev-v1.1.json
From the link given below, download the 'The Stanford Sentiment Treebank' dataset and save in $SST_DIR directory:
SST-2
From the link given below, download the 'MultiNLI Matched' dataset and save in $MNLI_DIR directory:
MNLI
SocialIQA data can be downloaded from the link given below:
SocialIQA
ATOMIC data can be downloaded from the link given below:
ATOMIC

Instructions to run code

To train the Question Answering model using the SQuAD dataset, run the following command from the main directory:

python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

This will save model checkpoints at path specified by --output_dir.

To train the Sentiment Analysis model using the SST-2 dataset, run the following command from the main directory:

python run_glue.py \
 --model_type bert \
 --model_name_or_path bert-base-cased \
 --task_name SST-2 \
 --do_train \
 --do_eval \
 --data_dir $SST_DIR/SST-2 \
 --max_seq_length 128 \
 --per_gpu_train_batch_size 32 \
 --learning_rate 2e-5 \
 --num_train_epochs 3.0 \
 --output_dir /tmp/SST-2/

To train the Natural Language Inference model using the MNLI dataset, run the following command from the main directory:

python run_glue.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --task_name MNLI \
  --do_train \
  --do_eval \
  --data_dir $MNLI_DIR/MNLI \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/MNLI/

To train the multitask model on SQuAD, SST-2 and MNLI, run the following command from the main directory:

python run_squad_sst_mnli.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_mnli \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --sst_data_dir $PATH_TO_SST \
  --mnli_data_dir $PATH_TO_MNLI \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

To train the vanilla BERT model using SocialIQA dataset, run the following command from the main directory:

python run_sqa.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --num_train_epochs 100 \
  --do_train \
  --do_eval \
  --evaluate_during_training \
  --overwrite_cache \
  --overwrite_output_dir \
  --output_dir output_sqa_run_100epochs/

To generate inferences from ATOMIC:
Run the setup scripts from the comet directory to acquire the pretrained model files from OpenAI, as well as the ATOMIC dataset:

bash scripts/setup/get_atomic_data.sh
bash scripts/setup/get_model_files.sh

Run the following script to pre-initialize a data loader for ATOMIC:

python scripts/data/make_atomic_data_loader.py

For running the ATOMIC experiment:
For whichever experiment # you set in config/atomic/changes.json (e.g., 0, 1, 2, etc.), run:

python src/main.py --experiment_type atomic --experiment_num #

To train the model using SocialIQA and inferences from commonsense, run the following command from the comet directory:

python run_sqa_cs.py \
--model_type bert \
--model_name_or_path bert-base-uncased \
--num_train_epochs 100 \
--do_train \
--do_eval \
--evaluate_during_training \
--overwrite_cache \
--overwrite_output_dir \
--output_dir output_sqa_run_100epochs/

For running the SQuAD to SocialIQA transfer learning experiment, first train the model on SQuAD (as given in step 1 above) and copy a checkpoint (we have used checkpoint-14000) from /tmp/debug_squad to the main directory. Run the following command from the main directory:

python run_sqa.py \
--model_type bert \
--model_name_or_path checkpoint-14000 \
--num_train_epochs 100 \
--do_train \
--do_eval \
--evaluate_during_training \
--overwrite_cache \
--overwrite_output_dir \
--output_dir output_sqa_run_100epochs/

Contact Us

Due to space constraints, we have not hosted the data and models/model checkpoints in this repository. If interested in running our code, feel free to get in touch with us by email and do share questions, comments and suggestions!

rohitgajawada/Common-Sense-QA