/qgen-workshop

ImplementAI Workshop on Deep NLP for Question Generation

Primary LanguageJupyter NotebookMIT LicenseMIT

Multi-task Question and Answer Generation

With the goal of building an end-to-end model, we ended up building a multi-task model to generate (question, answer) pairs from a document. We combine a few core concepts for text processing using neural networks to build our model.

See the notebook for an explanation of the model with an overview of the code.

Setup

  • Install Python 3.x (we recommend using Anaconda/Miniconda).
  • If on Windows, it is highly recommended to install Numpy and SciPy built with MKL support. There's two ways to do this depending on your package manager:
    • conda: conda create --name q-gen python=3.5 h5py numpy pandas scipy to pull the packages maintained by Continuum through Anaconda.
    • pip: download NumPy + SciPy with MKL support from here. Install with pip install <path_to_whl>.
  • Install requirements:
    • Non Windows: pip install -r requirements.txt.
    • Windows: pip install -r requirements.win.txt.
  • Download GloVe 100 dim embeddings from here and extract to the root of this repo.
  • Download NewsQA and process per these instructions. Put (dev|test|train).csv into the root of this repo.

Training

Prepare the data:

PYTHONPATH=".:$PYTHONPATH" python qgen/data.py

Train:

PYTHONPATH=".:$PYTHONPATH" python qgen/model.py

Loading in TensorBoard

tensorboard --logdir='log_dir'