/text_detoxification

The page with all info about text SOTA detoxification models and datasets.

MIT LicenseMIT

Text Detoxification

toxic text neutral paraphrase
this is scaring the s**t out of me. This is really scaring me.
calm the f**k down Please calm your nerves
its a crock of s**t , and you know it. It's senseless, you know it

This repo summarize all the information about Text Detoxification project. Here, you can find all dataset, evaluation setups, and SOTA models for text detoxification for Enlgish and Russian languages.

Main Idea

We are the first to address text detoxification task as seq2seq generation task. To achieve this, we collected the parallel corpora of pairs toxic<->non-toxic texts (See illustration). After that, we trained LM on this corpora to perform detoxification task. These models perform very good and definitely way better than previous unsupervised approaches.

Datasets

We release all the main ParaDetox datates as well as the results of crowdsourcing tasks:

  • The negatily marked samples from Paraphrasing task can be used to train classifier to detect if the sentence is detoxified in general or not;
  • The results from Content task can be used to train paraphrase detector between toxic and non-toxic texts and then to measure texts similarity.
  • The results from Toxicity task can be used as additional data for toxicity classification.

English

  1. EnParaDetox: 🤗 https://huggingface.co/datasets/s-nlp/paradetox
  2. EnParaDetox: Paraphrase Task Negative Results: 🤗 https://huggingface.co/datasets/s-nlp/en_non_detoxified
  3. EnParaDetox: Content Task Results: 🤗 https://huggingface.co/datasets/s-nlp/en_paradetox_content
  4. EnParaDetox: Toxicity Task Results: 🤗 https://huggingface.co/datasets/s-nlp/en_paradetox_toxicity

Russian

  1. RuParaDetox: 🤗 https://huggingface.co/datasets/s-nlp/ru_paradetox
  2. RuParaDetox: Paraphrase Task Negative Results: 🤗 https://huggingface.co/datasets/s-nlp/ru_non_detoxified
  3. RuParaDetox: Content Task Results: 🤗 https://huggingface.co/datasets/s-nlp/ru_paradetox_content
  4. RuParaDetox: Toxicity Task Results: 🤗 https://huggingface.co/datasets/s-nlp/ru_paradetox_toxicity

Models

  1. English SOTA: 🤗 https://huggingface.co/s-nlp/bart-base-detox
  2. Russian SOTA: 🤗 https://huggingface.co/s-nlp/ruT5-base-detox

Evaluation

Automatic evaluation always is separated into three parameters: (i) style transfer accuracy (STA) which is usually estimated by toxicity classifier; (ii) content similarity (SIM) which can be estimated either via cosine simiarity between embeddings or as a score from a classifier; (iii) fluency (FL) which can be estimated either via perplexity from LM or as a score from language acceptability classifier. The most recent evaluation setup for the languages:

English

  1. Toxicity classifier: 🤗 https://huggingface.co/s-nlp/roberta_toxicity_classifier
  2. Content similarity classifier: 🤗
  3. Fluency classifier: 🤗 https://huggingface.co/cointegrated/roberta-large-cola-krishna2020

Russian

  1. Toxicity classifier: 🤗 https://huggingface.co/IlyaGusev/rubertconv_toxic_clf
  2. Text embedder: 🤗 https://huggingface.co/sentence-transformers/LaBSE (russian part)
  3. Fluency classifier: 🤗

Demos

You can check our telegram bot and take out all the anger on it!

Github Pages

  1. The first version of Russian Texts Detoxifier: simple baselines, Russian condBERT, ruGPT-3.
  2. Unsupervised English Texts Detoxiifcation: English condBERT, ParaGedi.
  3. English ParaDetox: English ParaDetox dataset, the first supervised English detoxification SOTA bart-detox.
  4. Russian ParaDetox: competition RUSSE-2022 details, Russian ParaDetox dataset, the first supervised Russian detoxification SOTA ruT5-detox

Papers

  1. Logacheva, V.*, Dementieva, D.*, Ustyantsev, S., Moskovskiy, D., Dale, D., Krotova, I., ... & Panchenko, A. (2022, May). ParaDetox: Detoxification with Parallel Data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6804-6818). (*equal contribution) TL;DR EnParaDetox and EnDetox SOTA. [paper]
  2. Dementieva, D., Logacheva, V., Nikishina, I., Fenogenova, A., Dale, D., Krotova, I., ... & Panchenko, A. RUSSE-2022: Findings of the First Russian Detoxification Shared Task Based on Parallel Corpora. TL;DR RuParaDetox and RuDetox SOTA. [paper]
  3. Dementieva, D., Moskovskiy, D., Logacheva, V., Dale, D., Kozlova, O., Semenov, N., & Panchenko, A. (2021). Methods for detoxification of texts for the russian language. Multimodal Technologies and Interaction, 5(9), 54. TL;DR The first detoxification experiments on Russian, unsupervised, condBERT for TST. [paper]
  4. Dale, D., Voronov, A., Dementieva, D., Logacheva, V., Kozlova, O., Semenov, N., & Panchenko, A. (2021, November). Text Detoxification using Large Pre-trained Neural Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 7979-7996). TL;DR Unsupervised English detoxification based on condBERT and ParaGedi. [paper]
  5. Logacheva, V.*, Dementieva, D.*, Krotova, I., Fenogenova, A., Nikishina, I., Shavrina, T., & Panchenko, A. (2022, May). A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval) (pp. 90-101). (*equal contribution) TL;DR The problems in automatic evaluation of TST tasks. [paper]
  6. Moskovskiy, D., Dementieva, D., & Panchenko, A. (2022, May). Exploring Cross-lingual Text Detoxification with Large Multilingual Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (pp. 346-354). TL;DR Transferring detoxification knowledge between English and Russian. [paper]

Contact

Any question, suggestions, discussions to: Daryna Dementieva (dardem96@gmail.com)