This repository contains the implementation of the paper: "POINTER: Constrained Text Generation via Insertion-based Generative Pre-training", a progressive and non-autoregressive text generation pre-training approach.
Figure: Illustration of the generation process (blue arrow) of the proposed POINTER model. At each stage, the module generates either a
or a special
NOI
token
for each gap between two existing tokens . The generation stops when all the gaps predict NOI
. The data preparation process (orange arrow) reverses the above generative process.
The live demo can be found at here. Please expect delay and crash as it is running on a single GPU machine.
Please use the below commandlines to clone, install the requirements and load the Conda environment (Note that Cuda 10 is required):
sudo apt-get install -y make wget gzip bzip2 xz-utils zstd
conda env create -f LSP-linux.yml -n LSP
conda activate LSP
If you run this on an architecture other than Linux, please use LSP-generic.yml
instead of LSP-linux.yml
but please note that the generic one is not tested in all platform, so the stablity can not be gauranteed.
Link to the model and config files can be downloaded here.
To continue, please decompress the file and move the ckpt
folder into the main directory of this repo
tar -xzvf ckpt.tar.gz
Quick start (TL;DR): Run the demo in our repo as
./demo.sh
Decoding script:
Please put an test.key.txt
(see the input/test.key.txt
in this repo for an example) into the input
folder of this code, with \t
seperating the constraints. The generation can be done using following command:
conda activate LSP
python ./code/inference.py \
--pregenerated_data ./input \
--bert_model $model_path \
--output_dir $result_path \
The generation will be at the $result_path
folder.
If you use this code in your research, you can cite our paper:
@article{zhang2020pointer,
title={POINTER: Constrained Text Generation via Insertion-based Generative Pre-training},
author={Zhang, Yizhe and Wang, Guoyin and Li, Chunyuan and Gan, Zhe and Brockett, Chris and Dolan, Bill},
journal={arXiv preprint arXiv:2005.00558},
year={2020}
}