This is a repository to form a bengali data set scraping the bengali wikipedia. This is primarily done for the Mozzila Common Voice Bengali and will contain the বাংলা উইকিপেডিয়া(bengali wikipedia) for now. But it will contain sentences from other public domain in the future.
- create usable languge rules and a blacklist.
This is supposed to work with cv-sentence-extractor.
Clone common-voice-wiki-scraper repo:
git clone https://github.com/Common-Voice/common-voice-wiki-scraper.git
You need to download the WikiExtractor:
git clone https://github.com/attardi/wikiextractor.gitclone this repo
git clone https://github.com/mm-crj/bn-wiki-scraper.git
cd bn-wiki-scraperextract the text.tar.bz2 file.
then place the text folder from this repo in wikiextractor folder(cloned or downloaded).
Place the files from the rules folder in this repo in appropiate places in the common-voice-wiki-scraper folder.
then execute
cargo run -- extract -l bn -d ../wikiextractor/text/ >> ./sentences/wiki.bn.txt