/text-style-transfer-benchmark

Text style transfer benchmark

Text Style Transfer Benchmark

This benchmark is built for neural text style transfer. We are still collecting the relevant results and paper, if you want to add your own paper on this benchmark, feel free to contact us by sending mail to szha2609 [at] uni.sydney.edu.au or ykshi.1991 [at] foxmail.com, we will update your data ASAP.

Yelp Dataset

Dataset: Yelp Content Preservation Naturalness Transfer Intensity
Model WMD BLEU B-P B-R B-F1 N-A N-C N-D ACCU EMD
Language-Discriminator(yang2018) 0.1014 63.77 0.7292 0.7329 0.7314 0.5886 0.9074 0.6389 0.8940 0.8559
Struct(tian2018) 0.1224 61.98 0.7174 0.7220 0.7200 0.6205 0.9261 0.7002 0.8960 0.8574
StyleTrans-multi(dai2019) 0.1536 63.08 0.7145 0.7203 0.7175 0.6133 0.9102 0.6909 0.8730 0.8316
DualRL(luo2019) 0.1692 59.01 0.7125 0.6988 0.7057 0.5517 0.8996 0.6768 0.9050 0.8675
Texar(hu2018) 0.1921 57.82 -- -- -- 0.6934 0.9373 0.7066 0.8850 0.8429
StyleTrans-cond(dai2019) 0.2223 53.28 0.6205 0.6475 0.6341 0.6312 0.9109 0.6654 0.9290 0.8815
UnsuperMT(zhang2018) 0.2450 46.25 0.6060 0.6206 0.6134 0.5755 0.9040 0.6625 0.9770 0.9372
UnpairedRL(xu2018) 0.3122 46.09 0.4504 0.4709 0.4612 0.7136 0.9035 0.6493 0.5340 0.4989
DAR_Template(li2018) 0.4156 57.10 0.4970 0.5406 0.5185 0.6370 0.8984 0.6299 0.8410 0.7948
DAR_DeleteOnly(li2018) 0.4538 34.53 0.4158 0.4823 0.4490 0.6345 0.9072 0.5511 0.8750 0.8297
DAR_DeleteRetrieve(li2018) 0.4605 36.72 0.4268 0.4799 0.4534 0.6564 0.9359 0.5620 0.9010 0.8550
CAAE(shen2017) 0.5130 20.74 0.3585 0.3825 0.3710 0.4139 0.7006 0.5999 0.7490 0.7029
IMaT(jin2019) 0.5571 16.92 0.4750 0.4249 0.4501 0.4878 0.8407 0.6691 0.8710 0.8198
Multi_Decoder(fu2018) 0.5799 24.91 0.3117 0.3315 0.3223 0.4829 0.8394 0.6365 0.6810 0.6340
FineGrained-0.7(luo2019) 0.6239 11.36 0.4023 0.3404 0.3717 0.3665 0.7125 0.5332 0.3960 0.3621
FineGrained-0.9(luo2019) 0.6251 11.07 0.4030 0.3389 0.3713 0.3668 0.7148 0.5231 0.4180 0.3926
FineGrained-0.5(luo2019) 0.6252 11.72 0.3994 0.3436 0.3718 0.3608 0.7254 0.5395 0.3280 0.2985
BackTranslation(prabhumoye2018) 0.7566 2.81 0.2405 0.2024 0.2220 0.3686 0.5392 0.4754 0.9500 0.9117
Style_Emb(fu2018) 0.8796 3.24 0.0166 0.0673 0.0429 0.5788 0.9075 0.6450 0.4490 0.4119
DAR_RetrieveOnly(li2018) 0.8990 2.62 0.1368 0.1818 0.1598 0.8067 0.9717 0.7211 0.9610 0.9010
ARAE(zhao2018) 0.9047 5.95 0.1680 0.1478 0.1584 0.4476 0.8120 0.6969 0.8278 0.7880

Political Slant Dataset

Dataset:   Political Content Preservation Naturalness Transfer Intensity
Model WMD BLEU B-P B-R B-F1 N-A N-C N-D ACCU EMD
CAAE(shen2017) 0.4968 15.68 0.3217 0.3240 0.3230 0.2715 0.7052 0.7370 0.828 0.8259
DAR_DeleteOnly(li2018) 0.5000 30.76 0.3295 0.3932 0.3605 0.3155 0.8534 0.6490 0.958 0.9565
DAR(li2018) 0.5109 35.48 0.3352 0.3966 0.3649 0.3190 0.8472 0.7081 0.977 0.9747
DAR_RetrieveOnly(li2018) 0.7771 10.14 0.1590 0.1840 0.1709 0.3219 0.7854 0.7271 0.998 0.9960
ARAE(zhao2018) 1.0347 2.95 0.0203 0.0117 0.0158 0.3092 0.7763 0.7333 0.944 0.9412

Paper-news Title Dataset

Dataset:   Title Content Preservation Naturalness Transfer Intensity
Model WMD BLEU B-P B-R B-F1 N-A N-C N-D ACCU EMD
DAR_DeleteOnly(li2018) 0.8413 4.75 0.0939 0.0412 0.0677 0.3912 0.8374 0.4495 0.881 0.8687
DAR(li2018) 0.8567 5.04 0.0249 0.0212 0.0234 0.2462 0.7387 0.4625 0.933 0.9234
CAAE(shen2017) 0.9226 0.82 0.0067 -0.0099 -0.0008 0.2167 0.5627 0.4422 0.972 0.9612
DAR_RetrieveOnly(li2018) 0.9842 0.37 -0.0383 -0.0362 -0.0365 0.1490 0.5701 0.4261 0.995 0.9856
ARAE(zhao2018) 1.0253 0.00 -0.0447 -0.0539 -0.0486 0.2318 0.6061 0.4765 0.989 0.9782

Relevant Citation

@article{dai2019,
  title={Style transformer: Unpaired text style transfer without disentangled latent representation},
  author={Dai, Ning and Liang, Jianze and Qiu, Xipeng and Huang, Xuanjing},
  journal={arXiv preprint arXiv:1905.05621},
  year={2019}
}

@article{luo2019,
  title={A dual reinforcement learning framework for unsupervised text style transfer},
  author={Luo, Fuli and Li, Peng and Zhou, Jie and Yang, Pengcheng and Chang, Baobao and Sui, Zhifang and Sun, Xu},
  journal={arXiv preprint arXiv:1905.10060},
  year={2019}
}

@article{zhang2018,
	title="Style Transfer as Unsupervised Machine Translation",
	author="Zhirui {Zhang} and Shuo {Ren} and Shujie {Liu} and Jianyong {Wang} and Peng {Chen} and Mu {Li} and Ming {Zhou} and Enhong {Chen}",
	journal="arXiv preprint arXiv:1808.07894",
	notes="Sourced from Microsoft Academic - https://academic.microsoft.com/paper/2888173624",
	year="2018"
}

@article{xu2018,
	title="Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach",
	author="Jingjing {Xu} and Xu {Sun} and Qi {Zeng} and Xuancheng {Ren} and Xiaodong {Zhang} and Houfeng {Wang} and Wenjie {Li}",
	journal="arXiv preprint arXiv:1805.05181",
	notes="Sourced from Microsoft Academic - https://academic.microsoft.com/paper/2801454967",
	year="2018"
}

@inproceedings{li2018,
	title="Delete, retrieve, generate: A simple approach to sentiment and style transfer",
	author="Juncen {Li} and Robin {Jia} and He {He} and Percy {Liang}",
	booktitle="NAACL HLT 2018: 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
	volume="1",
	pages="1865--1874",
	year="2018"
}

@article{shen2017,
	title="Style transfer from non-parallel text by cross-alignment",
	author="Tianxiao {Shen} and Tao {Lei} and Regina {Barzilay} and Tommi S. {Jaakkola}",
	journal="Neural Information Processing Systems",
	pages="6830--6841",
	year="2017"
}

@article{jin2019,
  title={IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and Translation},
  author={Jin, Zhijing and Jin, Di and Mueller, Jonas and Matthews, Nicholas and Santus, Enrico},
  journal={arXiv preprint arXiv:1901.11333},
  year={2019}
}

@article{fu2018,
	title="Style transfer in text: Exploration and evaluation",
	author="Zhenxin {Fu} and Xiaoye {Tan} and Nanyun {Peng} and Dongyan {Zhao} and Rui {Yan}",
	journal="National Conference on Artificial Intelligence",
	pages="663--670",
	year="2018"
}


@article{prabhumoye2018,
	title="Style transfer through back-translation",
	author="Shrimai {Prabhumoye} and Yulia {Tsvetkov} and Ruslan {Salakhutdinov} and Alan W {Black}",
	journal="Meeting of the Association for Computational Linguistics",
	volume="1",
	pages="866--876",
	year="2018"
}

@inproceedings{zhao2018,
	title="Adversarially regularized autoencoders",
	author="Junbo Jake {Zhao} and Yoon {Kim} and Kelly {Zhang} and Alexander M. {Rush} and Yann {LeCun}",
	booktitle="ICML 2018: Thirty-fifth International Conference on Machine Learning",
	pages="9405--9420",
	year="2018"
}

@article{hu2018,
  title={Texar: A modularized, versatile, and extensible toolkit for text generation},
  author={Hu, Zhiting and Shi, Haoran and Tan, Bowen and Wang, Wentao and Yang, Zichao and Zhao, Tiancheng and He, Junxian and Qin, Lianhui and Wang, Di and Ma, Xuezhe and others},
  journal={arXiv preprint arXiv:1809.00794},
  year={2018}
}

@inproceedings{yang2018,
  title={Unsupervised text style transfer using language models as discriminators},
  author={Yang, Zichao and Hu, Zhiting and Dyer, Chris and Xing, Eric P and Berg-Kirkpatrick, Taylor},
  booktitle={Advances in Neural Information Processing Systems},
  pages={7287--7298},
  year={2018}
}

@article{tian2018,
  title={Structured content preservation for unsupervised text style transfer},
  author={Tian, Youzhi and Hu, Zhiting and Yu, Zhou},
  journal={arXiv preprint arXiv:1810.06526},
  year={2018}
}