This repository contains:
(1) The YASO evaluation dataset for targeted sentiment analysis (TSA).
(2) Code for evaluating the output of TSA systems on YASO.
(3) New! The TSA-MD dataset, described here.
(4) New! Domain labels added to the YASO evaluation datast, see the changelog for details.
Data: Some of the sentences annotated in YASO are taken from other datasets that cannot be re-distributed in clear text. To obtain the original texts please follow the instructions here.
Evaluation code:
Installation
Using pip:
pip install git+ssh://git@github.com/IBM/yaso-tsa.git#egg=yaso-tsa
Alternatively, you can first clone the code, and install the requirements:
1. git clone git@github.com:IBM/yaso-tsa.git
2. cd yaso-tsa/yaso_tsa
3. pip install -r requirements.txt
Running an evaluation
Use the module yaso_tsa.evaluate_tsa.
For example, run the following command from the main directory of the repository:
python -m yaso_tsa.evaluate_tsa --predictions_path tests/data/test_data.json --labels_path tests/data/test_labels.json
The expected output should be similar to:
[MainThread] 2021-09-13:16:37:15,137 INFO [evaluate_tsa.py:34] Loaded labeled data: <TsaLabels labeled: 4, sentences: 3>
[MainThread] 2021-09-13:16:37:15,190 INFO [evaluate_tsa.py:44] precision=0.6666666666666666
[MainThread] 2021-09-13:16:37:15,190 INFO [evaluate_tsa.py:44] recall=0.6666666666666666
[MainThread] 2021-09-13:16:37:15,190 INFO [evaluate_tsa.py:44] F1=0.6666666666666666
If you are using YASO in a publication, please cite the following paper:
Matan Orbach, Orith Toledo-Ronen, Artem Spector, Ranit Aharonov, Yoav Katz and Noam Slonim. 2021. YASO: A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews. EMNLP.
This project welcomes external contributions, if you would like to contribute please see further instructions here
Pull requests are very welcome! Make sure your patches are well tested. Ideally create a topic branch for every separate change you make. For example:
- Fork the repo
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Added some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
Major changes are documented here.
If you have any questions or issues you can create a new issue here.
This code is distributed under Apache License 2.0. If you would like to see the detailed LICENSE click here.
The YASO dataset was collected by Matan Orbach, Orith Toledo-Ronen, Artem Spector, Ranit Aharonov, Yoav Katz and Noam Slonim.
The evaluation code was written by Matan Orbach and Artem Spector.