Unofficial PyTorch implementation of "Step-unrolled Denoising Autoencoders for Text Generation"
Work-in-progress, more experiments soon.
Currently implements unconditional character-level text generation on the text8 dataset.
Requires x-transformers and ptpt to be installed.
Run SUNDAE training using config file cfg.toml:
python main.py --cfg-path cfg.toml
Sample text from trained model saved at ckpt.pt:
python main.py --cfg-path cfg.toml --resume ckpt.pt --sample --nb-samples 16
Other useful flags:
--seed # set RNG seed
--no-save # disable saving of checkpoints [False]
--no-cuda # disable the use of CUDA device [False]
--no-amp # disable the use of automatic mixed precision [False]
--nb-workers # set number of dataloader workers. [4]
- Add argmax-unrolled sampling
- Encoder-decoder translation experiments
- Pre-trained checkpoints
Step-unrolled Denoising Autoencoders for Text Generation
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, Aaron van den Oord
@misc{savinov2021stepunrolled,
title={Step-unrolled Denoising Autoencoders for Text Generation},
author={Nikolay Savinov and Junyoung Chung and Mikolaj Binkowski and Erich Elsen and Aaron van den Oord},
year={2021},
eprint={2112.06749},
archivePrefix={arXiv},
primaryClass={cs.CL}
}