code for our NAACL 2019 paper "BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis", COLING 2020 paper "Understanding Pre-trained BERT for Aspect-based Sentiment Analysis" and (draft code of) Findings of EMNLP 2020 "DomBERT: Domain-oriented Language Model for Aspect-based Sentiment Analysis".
We found that BERT domain post-training (e.g, 1 day of training) is an economic way to boost the performance of BERT, because it is much harder (e.g., 10 days of training) to learn a general knowledge shared across domains and, meanwhile, loosing the long-tailed domain-specific knowledge.
Code base for "Understanding Pre-trained BERT for Aspect-based Sentiment Analysis" is released.
Code base on huggingface transformers is under transformers, with more cross-domain models.
Preprocessing ABSA xmls organized into a separate rep.
Want to have post-trained models for other domains in reviews ? checkout a cross-domain review BERT or download from HERE.
A conversational dataset of RRC can be found here.
If you only care about ASC, a more formal code base can be found in a similar rep focusing on ASC.
**feedbacks are welcomed for missing instructions **
We focus on 3 review-based tasks: review reading comprehension (RRC), aspect extraction (AE) and aspect sentiment classification (ASC).
RRC: given a question ("how is the retina display ?") and a review ("The retina display is great.") find an answer span ("great") from that review;
AE: given a review sentence ("The retina display is great."), find aspects("retina display");
ASC: given an aspect ("retina display") and a review sentence ("The retina display is great."), detect the polarity of that aspect (positive).
E2E-ABSA: the combination of the above two tasks as a sequence labeling task.
And how a pre-trained BERT model on reviews be prepared for those tasks.
For post-training of NAACL 2019 paper, the code base is splited into two versions: transformers/ (instructions) and pytorch-pretrained-bert/ (instructions).
For analysis of pre-trained BERT model for ABSA (COLING 2020), see this instructions.
Please check corresponding instructions for details.
If you find this work useful, please cite as following.
@inproceedings{xu_bert2019,
title = "BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis",
author = "Xu, Hu and Liu, Bing and Shu, Lei and Yu, Philip S.",
booktitle = "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics",
month = "jun",
year = "2019",
}
@inproceedings{xu_understanding2020,
title = "Understanding Pre-trained BERT for Aspect-based Sentiment Analysis",
author = "Xu, Hu and Shu, Lei and Yu, Philip S. and Liu, Bing",
booktitle = "The 28th International Conference on Computational Linguistics",
month = "Dec",
year = "2020",
}