splitter for generating transliteration corpus
Description
- This is Python script that uses the text file generated by 'Wikipedia Parallel Title Extractor - https://github.com/clab/wikipedia-parallel-titles' as an input.
- This script process the input text file (mentioned above) to generate a parallel corpus.
- Output of this script (parallel corpus) can be used to train transliteration model on MOSES.
Author
- Moodser Hussain
- COMSATS University Islamabad, Lahore Campus
- Email: moodser.hussain@gmail.com
Acknowledgement
Special thanks to Dr. Rao Muhammad Adeel Nawab and Sir Muhammad Sharjeel for their continous support.
Usage
- Download the script file (splitter.py)
- Copy the input file (generated by wikipedia parallel title script) in same directory
- run the terminal/cmd command 'python splitter.py '
- Two output files will be generated for each language seperately.
Caution
- This Script is tested on English-Urdu parallel titles extracted from https://dumps.wikimedia.org/urwiki/20180801/ using https://github.com/clab/wikipedia-parallel-titles
- Python version 3.6 was used for testing this script.