DeepOffense provides state-of-the-art transformer models for multilingual offensive language identification.
You first need to install PyTorch. THe recommended PyTorch version is 1.5. Please refer to PyTorch installation page regarding the specific install command for your platform.
When PyTorch has been installed, you can install from source by cloning the repository and running:
git clone https://github.com/TharinduDR/DeepOffense.git
cd DeepOffense
pip install -r requirements.txt
Examples are included in the repository but are not shipped with the library. Please refer the examples directory for the examples. Each directory in the examples folder contains different languages.
English offensive language detection pre-trained model trained with XLM-R large model on OffensEval data can be downloaded using this link.
Once downloading it and unzipping it, they can be loaded easily. To see how to begin the training process please refer the examples directory
model = ClassificationModel("xlmroberta", "path", use_cuda=torch.cuda.is_available())
Please consider citing us if you use the library.
@inproceedings{ranasinghe-etal-2020-multilingual,
title = "Multilingual Offensive Language Identification with Cross-lingual Embeddings",
author = "Ranasinghe, Tharindu and
Zampieri, Marcos,
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
month = nov
year = "2020",
}
Citation for the Malayalam specific paper,
@inproceedings{ranasinghe-etal-2020-wlv,
title={WLV-RIT at HASOC 2020: Offensive Language Identification in Code-switched Texts},
author={Ranasinghe, Tharindu and Zampieri, Marcos},
year={2020},
booktitle={Proceedings of FIRE}
}