The settings for Hyper-Parameters
Opened this issue · 4 comments
Hi, I'm wondering what are the best settings for Hyper-Parameters α，β1， β2， γ in the overall objective？And why did you select β from{0.5,1,1.5,3,5,10,15} ?
Sorry for late reply. The hyper-parameters are a little sensitive and required to be tuned on the given dataset (because it is a unsupervised task). We tune these hyper-parameters according to the model performance on the validation set.
A simple rule is: a larger
@hzhwcmhf Thank you for your kind reply, I've followed your instruction to tune and then it comes to evaluation step. And I'm wondering how you select and preprocess the data for the classifier and the ppl model. Could you please share the detailed setting and the script? Thank you so much.
In your readme.md, you mentioned that in the classifier training, some samples from the original test set are in the train.txt, and some of them are in the test.txt. Could you please give me more hints why you apply this setting and share more details.
@FayeXXX We randomly sample some data from the original training and test set to form the training set for the classification and language models. The final training set is "yelp/classifier|languagemodel/train.txt", and the test set is "yelp/classifier|languagemodel/test.txt".
Here is the reason why we sample some data from the original test set:
Since this is an unsupervised task, the training set only contains unparallel data, so the classifiers never see two sentences having similar contents but different sentiments. Therefore, it can be difficult for the classifier to distinguish the transferred samples (written by human or generated by models) from the original ones. For example, "they will even deliver to the school !" and "they will not deliver to the school !" has opposite sentiments, but a small classifier could easily ignore the not word and label them with the same sentiment. It is also called as an "adversarial" sample. (Here is a paper to construct adversarial samples http://aclanthology.lst.uni-saarland.de/2020.emnlp-main.498.pdf)
The simplest way to tackle the problem is to do adversarial training (A survey paper). That is, train the model with some adversarial samples. In our case, we add some human-written "adversarial" samples (from the test set) to improve the performance against "the adversarial attack", leading to a better correlation with human evaluation.