/question_answering

Question Answering task using Deep Learning on SQuAD dataset

Primary LanguagePython

Question Answering with SQuAD

The goal of this project was for me to get familiar with the Question Answering task, a very active topic in NLP research.

To this end, I implemented a Bidirectional Attention Flow neural network as a baseline, improving Chris Chute's model implementation, adding word-character inputs as described in the original paper.

I found this project very useful from a learning perspective so I highly recommend you to dig into the code and work on improving this baseline.

You can reproduce the work following the Set-Up section, and potentially (recommended!) train the model on a single GPU setting the cuda variable in config.py to True.

Model Architecture

BiDAF Architecture Source: BiDAF paper

Code Organization

├── config.py          <- Configuration file with data directories and hyperparamters to train the model
├── data_loader.py     <- Define an iterator who collects batches of data to train the model
├── eval.py            <- Evaluate the model on a new pair of (context, question)
├── layers.py          <- Define the various layers to be used by the main BiDAF model
├── make_dataset.py    <- Download the SquAD dataset and pre-process the data for training
├── model.py.          <- Define the BiDAF model architecture
├── requirements.txt   <- Required Python libraries to build the project
├── test.py            <- Test the performance of a trained model on the DEV dataset
├── train.py           <- Train a model using the TRAIN dataset only
├── utils.py           <- Group a bunch of useful functions to process the data

Results

Loss and Metrics

Exact-Match and F1 Score on Validation set after training:

EM F1
0.64 0.75

Set-Up

  • Clone the repository
  • Create a directory for your experiments, logs and model weights: mkdir output
  • Download GloVE word vectors: https://nlp.stanford.edu/projects/glove/
  • Modify the config.py file to set up the paths where your GloVE, SquAD and models will be located
  • Create a Python virtual environment, source to it: mkvirualenv qa-env ; workon qa-env if you use virtualenvwrapper
  • Install the dependencies: pip install -r requirements.txt ; python -m spacy download en
  • Run python make_dataset.py to download SquAD dataset and pre-process the data
  • Run python train.py to train the model with hyper-parameters found in config.py
  • Run python test.py to test the model EM and F1 scores on Dev examples
  • Play with eval.py to answer your own questions! :)

Next Steps

  • set up a variable to choose between training the model with word only VS word + characters
  • collect the moving average of the weights during training and use them during testing
  • add the ability to train the model on multiple GPUs, and offer half-precision training to speed-up the training
  • improve this baseline using pre-training encoding such as BERT, and/or set-up a multi-task learning pipeline to jointly learn to answer questions together with another closely related NLP task.

Resources