/arabic-text-diacritization

Benchmark Arabic text diacritization dataset

Primary LanguagePythonMIT LicenseMIT

Arabic Text Diacritization

This repository contains the dataset, helpers, and systems comparison for our paper on Arabic Text Diacritization:

"Arabic Text Diacritization Using Deep Neural Networks", Ali Fadel, Ibraheem Tuffaha, Bara' Al-Jawarneh, and Mahmoud Al-Ayyoub, ICCAIS 2019.

Files

dataset

  • train.txt - Contains 50,000 lines of diacritized Arabic text which can be used as training dataset
  • val.txt - Contains 2,500 lines of diacritized Arabic text which can be used as validation dataset
  • test.txt - Contains 2,500 lines of diacritized Arabic text which can be used as testing dataset

helpers

  • constants
    • ARABIC_LETTERS_LIST.pickle - Contains list of Arbaic letters
    • CLASSES_LIST.pickle - Contains list of all possible classes
    • DIACRITICS_LIST.pickle - Contains list of all diacritics
  • count_characters.py - Counts the number of Arabic letters and diacritics in a file
  • count_fathatan.py - Counts the number of fathatan occurrences before and after Alif in all files from a folder
  • diacritization_stat.py - Calculates DER and WER using the gold data and the predicted output
  • diacritics_rate_extractor.py - Keeps lines with p% diacritics to Arabic characters rate or more in all files from a folder
  • file_lookup.py - Searches for a line in all files from a folder
  • fix_fathatan.py - Changes after-Alif fathatan to before-Alit fathatan in a file
  • remove_diacritics.py - Removes diacritics from a file
  • transliteration.py - Converts a file from Arabic text to Buckwalter transliteration and vice-versa
  • pre_process_tashkeela_corpus.ipynb - Pre-process Tashkeela Corpus data

existing_systems

  • ali-soft - Contains some bugs that exist in Ali-Soft system
  • farasa - Contains Farasa system output, fixed output, and DER/WER statistics
  • harakat - Contains Harakat system testing script, output, fixed output, and DER/WER statistics
  • madamira - Contains MADAMIRA system output, fixed output, and DER/WER statistics
  • mishkal - Contains Mishkal system output, fixed output, and DER/WER statistics
  • shakkala - Contains Shakkala system data splitting script, output, fixed output, and DER/WER statistics
  • tashkeela_model - Contains Tashkeela-Model system output, fixed output, and DER/WER statistics for each n-gram model provided by them

Note: All codes in this repository tested on Ubuntu 18.04

Contributors

  1. Ali Hamdi Ali Fadel.
  2. Ibraheem Tuffaha.
  3. Bara' Al-Jawarneh.
  4. Mahmoud Al-Ayyoub.

License

The project is available as open source under the terms of the MIT License.