tesseract-ocr/langdata

Language request: Kurdish-Kurmanji

brandones opened this issue · 22 comments

brandones commented

It's the most popular dialect of Kurdish, written in a Latin alphabet with a few diacritics.

Shreeshrii commented
brandones commented

The data in both kur and kur_ara is in Arabic script, neither is in a Latin script.

Shreeshrii commented
Shreeshrii commented
brandones commented

Oh interesting! I will give that a try, thank you.

brandones commented

Yes, it's definitely trained on Kurmanji. It's doing a great job OCRing Kurmanji text. This is wonderful, thank you!

Shreeshrii commented

Thank you for your feedback.

We need to change the traineddata name so that it correctly reflects the language.

What would you suggest?

kur
kur_lat

Any other official language code for kurmanji.

brandones commented

Sure! My recommendations:

Don’t use kur for anything.

Depends on if you’re more interested in the script or the dialect. There are a few Kurdish languages using each of the scripts, the Latin-based one and the Persian-based one. Historically Cyrillic and Armenian scripts have been used also. I’m guessing the language is itself important though.

The ISO 639-3 code for Kurmanji (aka Northern Kurdish) is kmr.

It wouldn’t be at all unreasonable to prefix it as kur_kmr, and refer to it as Kurdish Kurmanji. Or just list kmr as Kurdish Kurmanji.

Similarly, I’d recommend using ckb or kur_ckb for the one with the Arabic-based script, Sorani.

Shreeshrii commented

Thanks.

@jbreiden @theraysmith what is your recommendation?

At a minimum kur_ara should be changed as it doesn't have Arabic in it.

Shreeshrii commented

tesseract-ocr/tessdata_fast#14 (comment)

OK. I think we should follow the suggestion by @amitdo, since it is in line with the way tesseract names other languages.

amitdo commented

Shree, please send PRs.

brandones commented

In case it’s not clear, this choice impedes the development of Tesseract support for Zaza, Gorani/Horami, and Southern Kurdish, each of which will require quite different dictionaries.

Shreeshrii commented

@brandones Yes, that will require different dictionaries. As of now I am not sure exactly which language dictionary is being used.

I am attaching a zip file with two traineddatas for Kurdish in Arabic script - Sorani.

If you are familiar with the script, please review and provide feedback as to accuracy and also whether the word frequency list is appropriate. (or refer to someone who can provide that feedback). Thanks!

kur_ara.traineddata.zip

Shreeshrii commented

https://en.wikipedia.org/wiki/Kurdish_languages

ISO 639-3 kur – inclusive code
Individual codes:
ckb – Central Kurdish
kmr – Northern Kurdish
sdh – Southern Kurdish

ckb - Sorani (Arabic/Persian script).
kmr - Kurmanji (Latin script)

keyochali commented

I want to improve and work on Kurdish Kurmanji and Sorani for both Latin an Arabic scripts.
how can I do that?
what is needed to be done?
how can I submit it?

Shreeshrii commented

Sorani in Arabic script is not available in tessdata_best and tessdata_fast.

https://github.com/tesseract-ocr/langdata_lstm/tree/master/kur
has minimal language data available for it.

You will need to identify unicode fonts and training text for the same before running any training.

keyochali commented

thank you so much
I just tested the kmr.traineddata

but for the sorani in Arabic script where can I train it?
what data is needed except for the text?

I mean do I need to do boxing some images?
can you show me an example of training in tessercat?
is there any documentation for training it?

keyochali commented

wonderful
thanks a lot

keyochali commented

how can I do it in windows
when tesseract is installed it has the training tools with it
how can I use them
is there any python script to do it?
what is a textilne(a word? or the line?)