/t5-question-generation

t5qg, a python library to finetune T5 on question generation and provide API to host the model prediction. For the model training, we rely on the multitasking objective where the models are optimized for the question answering and the answer extraction in addition to the question generation following huggingface tutorial.

Primary LanguagePythonOtherNOASSERTION

T5 Question Generation

t5qg is a python library to finetune T5 on question generation and provide API to host the model prediction. For the model training, we rely on the multitasking objective where the models are optimized for the question answering and the answer extraction in addition to the question generation following huggingface tutorial.

Get Started 🚀

git clone https://github.com/asahi417/t5-question-generation
cd t5-question-generation
pip install .

Model Training/Evaluation

CLI

  • Model Training
t5qg-train -c ckpt/test -m google/mt5-small -d squad

run t5qg-train -h to display all the options.

  • Model Evaluation (Get metric with nlg-eval to assess the model)
t5qg-eval -m ckpt/test/epoch_10/ -e ckpt/test/epoch_10/eval

Python

  • Model Training
import t5qg
trainer = t5qg.Trainer(checkpoint_dir='ckpt/test', model='t5-small', epoch=5)
trainer.train()
  • Model Evaluation (Get metric with nlg-eval to assess the model)
import t5qg
t5qg.evaluate_qg(checkpoint_dir='ckpt/test/epoch_5')

Rest API

We provide a rest API which hosts the model inference.

  • From Command Line
uvicorn app:app --reload --port 80
  • Run with Docker
docker build -t t5qg/app:latest .
docker run -p 80:80 t5qg/app:latest

Swagger UI is available at http://127.0.0.1:80/docs. Model can be specified by providing the model alias on huggingface modelhub or the path to the checkpoint file to the environment variable MODEL (as default we use asahi417/question-generation-squad-t5-small).

QG Model Cards

Following models are available via the transformers modelhub. All models are trained over SQuAD for question generation where the data split follows Du, et al 2017 and Du, et al 2018. For each model, we add the link which includes BLEU-n, ROUGE, METEOR, and CIDEr produced by t5qg-eval.

Model Name Description BLEU 4 ROUGE L Other Metrics
asahi417/question-generation-squad-t5-small T5 small model trained on multitask loss 15.1 36.4 metric
asahi417/question-generation-squad-t5-base T5 base model trained on multitask loss 17.9 40.2 metric
asahi417/question-generation-squad-t5-large T5 large model trained on multitask loss 18.5 41.0 metric
asahi417/question-generation-squad-mt5-small mT5 small model trained on multitask loss 10.5 30.1 metric
asahi417/question-generation-squad-bart-base BART base model trained on QG task only 16.2 38.8 metric
asahi417/question-generation-squad-bart-large BART large model trained on QG task only 15.5 38.6 metric