coppercitylabs/uzbek-tokenizers

Various tokenizers for the Uzbek language

Uzbek Tokenizers

tokenizer models for the Uzbek language

Requirements

pip install -r requirements.txt

Sentence tokenizer

Trained using the NLTK Punkt tokenizer.

Example usage:

python scripts/sentence-tokenize.py