optimaize/language-detector

Norwegian profile: Bokmål or Nynorsk?

Closed this issue · 2 comments

fabiankessler commented

Norwegian is a macro language. The written standards are

  • Bokmål iso 639-1 "nb"
  • Nynorsk iso 639-1 "nn"

90% of all publications are in Bokmål, the remaining 10% in Nynorsk.

Technically, the code "no" may not be used in this context. It is not clear which it is.

I am pretty sure that the training text was from one standard only, and that it was Bokmål. But without having the original training text info I would not be able to tell (based on n-grams). A Norwegian might know the differences...

So the only way to fix this is to either get the original training text, or to create a new profile. Or 2 profiles to distinguish them.

janhoy commented

You are right. But most of the applications I have used detection for only care about detecting whether we have Norwegian, English or Chinese etc, not the "sub" language, so the ISO-639 macro language code "no" is good enough for many. Besides, Arabic is also a macro language, but do people care wether it is Tunisian or Egyptian arabic or any of the other 30 forms (according to https://en.wikipedia.org/wiki/ISO_639_macrolanguage)? I don't know.

So if someone create (good enough) detectors for individual languages like nb and nn, maybe the library should also be able to return both the language code and the macro language code?

fabiankessler commented

Thanks Jan for your comments. You're from Oslo, and in the search business, so your opinion is very relevant.

I see a small difference between Norwegian and Arabic.

Starting from the English Wikipedia article about the Eiffel Tower https://en.wikipedia.org/wiki/Eiffel_Tower you can get to the Arabic one https://ar.wikipedia.org/wiki/%D8%A8%D8%B1%D8%AC_%D8%A5%D9%8A%D9%81%D9%84 and yes there is just one for all Arabic variations.

For Norwegian we have 2:
https://no.wikipedia.org/wiki/Eiffelt%C3%A5rnet
https://nn.wikipedia.org/wiki/Eiffelt%C3%A5rnet

Wikipedia is cheating, they label the first "Norsk bokmål" but use the (common) macro language code "no" instead of "nb".

This is similar to what we have with German:
https://de.wikipedia.org/wiki/Eiffelturm
https://als.wikipedia.org/wiki/Eiffelturm
Just that in German the code "de" has a unique meaning.

I guess that this answers the question:
The language profile we have for "Norwegian" must have been created from plain Bokmål content. Just like the language profile for German used "de" pages and no other writing forms or dialects.

Then this is good enough. It's documented now. It matches the Wikipedia use of the language. And if one ever needed to detect Nynorsk separately, then he could create a profile for it based on Wikipedia "nn" content.