factoid_QA_with_distant_spervision

Codes for Factoid Question Answering With Distant Supervision.

I am cleaning the codes for uploading, and some description should be added.

Requirements

GPU and CUDA 8 are required
python >=3.5
pytorch 0.3.0
pandas
msgpack
spacy 1.x
cupy
pynvrtc
jieba

Download Data

Please download data files from google drive, and put the files under the "dat" file. Specifically, download these four files,

questions_dis_data_150htmls_using_abstext.txt
triple_weight_by_search.txt
new_mined_paraphrase0124.txt
WebQA.v1.0.tar.gz   # is it proper to upload this dataset?

Then unzip the WebQA data with tar -zxvf WebQA.v1.0.tar.gz.

Model training

Train the model via runing

cd DSRC
mkdir logs
python train_model.py

Please refer to parameters.py for configuration details, where train_idx is consponding to different experimental configurations in the paper.

Automatic training data generation via distant supervision

Besides the generated training data, we also released the data used to generate the training data, training sample selection and ming the distant paraphrases.

Training data generation via distant supervision

Coming soon.

Training sample selection and distant paraphrase minging

Credits

Autor of sru: Tao Lei.

Author of the Document Reader model: Danqi Chen.

Author of the original Pytorch implementation: Runqi Yang.

Most of the pytorch model code is borrowed from Facebook/ParlAI under a BSD-3 license.

yakzan/factoid_QA_with_distant_spervision