Weblate not outputing Android <plurals> fields properly, causing app crashes
Closed this issue · 18 comments
With Hebrew, the user interface shows One and Other as the options:
But the strings.xml then has One and Two filled out, then Many and Other blank:
<plurals name="details_last_update_weeks">
<item quantity="one">עודכן לפני שבוע</item>
<item quantity="two">עודכן לפני %1$d שבועות</item>
<item quantity="many"></item>
<item quantity="other"></item>
</plurals>An empty Other string can cause crashes. Especially because we remove blank strings, since they cause other problems. Weblate should ensure that if any item of <plurals> has been translated, then <item quantity="other"> must be translated. Here's the Android source reference on the requirement for other:
https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/content/res/ResourcesImpl.java#268
Server configuration
many, two352 <string name="updates__hide_updateable_apps">הסתרת יישומונים</string> 353 <string name="updates__show_updateable_apps">הצגת יישומונים</string> 354 355 <plurals name="updates__download_updates_for_apps"> 356 <item quantity="one">הורדת עדכון ליישומון אחד.</item> 357 <item quantity="other">הורדת עדכון ל־%1$d יישומונים.</item>../../src/main/res/values-iw/strings.xml:355: For locale "iw" (Hebrew) the following quantities should also be defined:
many, two352 <string name="updates__hide_updateable_apps">הסתרת יישומונים</string> 353 <string name="updates__show_updateable_apps">הצגת יישומונים</string> 354 355 <plurals name="updates__download_updates_for_apps"> 356 <item quantity="one">הורדת עדכון ליישומון אחד.</item> 357 <item quantity="other">הורדת עדכון ל־%1$d יישומונים.</item>../../src/main/res/values-he/strings.xml:437: For locale "he" (Hebrew) the following quantities should also be defined:
many, two434 <string name="notification_content_single_downloading_update">מתבצעת הורדת עדכון עבור „%1$s“…</string> 435 <string name="notification_content_single_installing">ההתקנה של „%1$s“ מתבצעת כעת…</string> 436 <string name="notification_content_single_installed">ההתקנה הצליחה</string> 437 <plurals name="notification_summary_updates"> 438 <item quantity="one">עדכון אחד</item> 439 <item quantity="other">%1$d עדכונים</item>../../src/main/res/values-iw/strings.xml:437: For locale "iw" (Hebrew) the following quantities should also be defined:
many, two434 <string name="notification_content_single_downloading_update">מתבצעת הורדת עדכון עבור „%1$s“…</string> 435 <string name="notification_content_single_installing">ההתקנה של „%1$s“ מתבצעת כעת…</string> 436 <string name="notification_content_single_installed">ההתקנה הצליחה</string> 437 <plurals name="notification_summary_updates"> 438 <item quantity="one">עדכון אחד</item> 439 <item quantity="other">%1$d עדכונים</item>../../src/main/res/values-he/strings.xml:441: For locale "he" (Hebrew) the following quantities should also be defined:
many, two438 <item quantity="one">עדכון אחד</item> 439 <item quantity="other">%1$d עדכונים</item> 440 </plurals> 441 <plurals name="notification_summary_installed"> 442 <item quantity="one">יישומון הותקן</item> 443 <item quantity="other">%1$d יישומונים הותקנו</item>+ 11 More Occurrences...
many, two438 <item quantity="one">עדכון אחד</item> 439 <item quantity="other">%1$d עדכונים</item> 440 </plurals> 441 <plurals name="notification_summary_installed"> 442 <item quantity="one">יישומון הותקן</item> 443 <item quantity="other">%1$d יישומונים הותקנו</item>../../src/main/res/values-he/strings.xml:454: For locale "he" (Hebrew) the following quantities should also be defined:
many, two451 <string name="notification_title_summary_installed">הותקן בהצלחה</string> 452 <string name="tts_category_name">קטגוריה %1$s</string> 453 454 <plurals name="tts_view_all_in_category"> 455 <item quantity="one">הצגת יישומון %1$d בקטגוריה %2$s</item> 456 <item quantity="other">הצגת כל %1$d היישומונים בקטגוריה %2$s</item>../../src/main/res/values-iw/strings.xml:454: For locale "iw" (Hebrew) the following quantities should also be defined:
many, two451 <string name="notification_title_summary_installed">הותקן בהצלחה</string> 452 <string name="tts_category_name">קטגוריה %1$s</string> 453 454 <plurals name="tts_view_all_in_category"> 455 <item quantity="one">הצגת יישומון %1$d בקטגוריה %2$s</item> 456 <item quantity="other">הצגת כל %1$d היישומונים בקטגוריה %2$s</item>../../src/main/res/values-he/strings.xml:460: For locale "he" (Hebrew) the following quantities should also be defined:
many, two457 </plurals> 458 459 <string name="details_last_updated_today">עודכן היום</string> 460 <plurals name="details_last_update_days"> 461 <item quantity="one">עודכן לפני יום</item> 462 <item quantity="other">עודכן לפני %1$d ימים</item>../../src/main/res/values-iw/strings.xml:460: For locale "iw" (Hebrew) the following quantities should also be defined:
many, two457 </plurals> 458 459 <string name="details_last_updated_today">עודכן היום</string> 460 <plurals name="details_last_update_days"> 461 <item quantity="one">עודכן לפני יום</item> 462 <item quantity="other">עודכן לפני %1$d ימים</item>../../src/main/res/values-he/strings.xml:464: For locale "he" (Hebrew) the following quantities should also be defined:
many, two461 <item quantity="one">עודכן לפני יום</item> 462 <item quantity="other">עודכן לפני %1$d ימים</item> 463 </plurals> 464 <plurals name="details_last_update_weeks"> 465 <item quantity="one">עודכן לפני שבוע</item> 466 <item quantity="other">עודכן לפני %1$d שבועות</item>../../src/main/res/values-iw/strings.xml:464: For locale "iw" (Hebrew) the following quantities should also be defined:
many, two461 <item quantity="one">עודכן לפני יום</item> 462 <item quantity="other">עודכן לפני %1$d ימים</item> 463 </plurals> 464 <plurals name="details_last_update_weeks"> 465 <item quantity="one">עודכן לפני שבוע</item> 466 <item quantity="other">עודכן לפני %1$d שבועות</item>../../src/main/res/values-he/strings.xml:468: For locale "he" (Hebrew) the following quantities should also be defined:
many, two465 <item quantity="one">עודכן לפני שבוע</item> 466 <item quantity="other">עודכן לפני %1$d שבועות</item> 467 </plurals> 468 <plurals name="details_last_update_months"> 469 <item quantity="one">עודכן לפני חודש</item> 470 <item quantity="other">עודכן לפני %1$d חודשים</item>../../src/main/res/values-iw/strings.xml:468: For locale "iw" (Hebrew) the following quantities should also be defined:
many, two465 <item quantity="one">עודכן לפני שבוע</item> 466 <item quantity="other">עודכן לפני %1$d שבועות</item> 467 </plurals> 468 <plurals name="details_last_update_months"> 469 <item quantity="one">עודכן לפני חודש</item> 470 <item quantity="other">עודכן לפני %1$d חודשים</item>../../src/main/res/values-he/strings.xml:472: For locale "he" (Hebrew) the following quantities should also be defined:
many, two469 <item quantity="one">עודכן לפני חודש</item> 470 <item quantity="other">עודכן לפני %1$d חודשים</item> 471 </plurals> 472 <plurals name="details_last_update_years"> 473 <item quantity="one">עודכן לפני שנה</item> 474 <item quantity="other">עודכן לפני %1$d שנים</item>../../src/main/res/values-iw/strings.xml:472: For locale "iw" (Hebrew) the following quantities should also be defined:
many, two469 <item quantity="one">עודכן לפני חודש</item> 470 <item quantity="other">עודכן לפני %1$d חודשים</item> 471 </plurals> 472 <plurals name="details_last_update_years"> 473 <item quantity="one">עודכן לפני שנה</item> 474 <item quantity="other">עודכן לפני %1$d שנים</item>
Category: Correctness:Messages
This lint check looks at each translation of a
<plural> and makes sure that all the quantity strings considered by the given language are provided by this translation.For example, an English translation must provide a string for
quantity="one". Similarly, a Czech translation must provide a string for quantity="few".
To suppress this error, use the issue id "MissingQuantity" as explained in the Suppressing Warnings and Errors section.
Apparently Gettext is not following CLDR plurals here and it causes problems in this case. Weblate is based on Gettext ones, where there are just two plural forms (n != 1), while CLDR (and Android) use four (one, two, many, other). This will be tricky to address...
As a temporary workaround, maybe you could just stick the Other value into Two and Many?
It seems that some languages are including more than One/Other. For example, Russian has One, Few, and Many, but is missing Other:
<plurals name="details_last_update_days">
<item quantity="one">Обновлено %1$d день назад</item>
<item quantity="few">Обновлено %1$d дня назад</item>
<item quantity="many">Обновлено %1$d дней назад</item>
</plurals>And Slovak includes One, Few, Many, and Other, but Many is apparently not supposed to be there:
<plurals name="updates__download_updates_for_apps">
<item quantity="one">Stiahnuť aktualizáciu pre %1$d aplikáciu.</item>
<item quantity="few">Stiahnuť aktualizácie pre %1$d aplikácie.</item>
<item quantity="many">Stiahnuť aktualizácie pre %1$d aplikácií.</item>
<item quantity="other">Stiahnuť aktualizácie pre %1$d aplikácií.</item>
</plurals>Serbian seems to have the correct plurals (One, Few, Other):
<plurals name="tts_view_all_in_category">
<item quantity="one">Прикажи %1$d апликацију у категорији %2$s</item>
<item quantity="few">Прикажи све %1$d апликације у категорији %2$s</item>
<item quantity="other">Прикажи свих %1$d апликација у категорији %2$s</item>
</plurals>This is problem only for languages where CLDR plurals do not match the ones commonly used by Gettext. I guess there won't be much of such languages...
PS: Slovak should have one/few/many/other, see http://www.unicode.org/cldr/charts/29/supplemental/language_plural_rules.html#sk.
Thinking more about this we really have to add plural definitions per translation, which would be based on file format and fallback to language definition if nothing is defined for file format or within the file. This way we could also address different plural forms being used for one language (such as #901).
Some formats will be easy to support properly:
- Use CLDR rules for Android (they can be probably extracted from Babel)
- Use
Plural-Formsheader for Gettext PO
Others might be a bit tricky and we will probably stick to current language defaults:
- XLIFF plurals are usually based on Gettext ones without mentioning the file format
- Qt TS rules themselves are not documented
Note: The Babel rules are not same as Gettext ones we're currently using, so it might need some more tweaks:
>>> import babel
>>> l = babel.Locale('cs')
>>> l.plural_form.tags
frozenset(['few', 'one', 'many'])
>>> l.plural_form.rules
{'few': 'i in 2..4 and v in 0', 'one': 'i in 1 and v in 0', 'many': 'v not in 0'}
>>> l.plural_form.abstract
[('few', ('and', (('relation', ('in', (u'i', ()), ('range_list', [(('value', (2,)), ('value', (4,)))]))), ('relation', ('in', (u'v', ()), ('range_list', [(('value', (0,)), ('value', (0,)))])))))), ('many', ('not', (('relation', ('in', (u'v', ()), ('range_list', [(('value', (0,)), ('value', (0,)))]))),))), ('one', ('and', (('relation', ('in', (u'i', ()), ('range_list', [(('value', (1,)), ('value', (1,)))]))), ('relation', ('in', (u'v', ()), ('range_list', [(('value', (0,)), ('value', (0,)))]))))))]
The variables are defined in the specification, there is also tool to convert them (written in PHP): https://github.com/mlocati/cldr-to-gettext-plural-rules
After digging into this for few hours (trying to add CLDR plurals to Weblate), now I finally understand where the problem is - it's not that the CLDR plural rules would be that different (the differences are there, but they are limited to few languages).
The problem lies a bit deeper:
Translate-toolkit always adds other to the plurals:
It might sound weird, but apparently the root cause is in Babel not returning other for most of the languages it should. There is even bug reported for similar thing since April 21, see python-babel/babel#495
This should be fixed by following PR which replaces Babel usage in translate-toolkit by own plurals data: translate/translate#3762
It would be really great if somebody with Android knowledge (eg. @eighthave , @TobiGr, @theScrabi ) could check if the tags are correct there - I was not able to find some official docs for Android covering this and the CLDR is still a bit magic for me as a result I'm not sure if the table I've generated correct (I've checked few languages which I know were problematic and they look correct).
I don't quite understand what I should check. I looked at this diff of 5d853aa and I have no idea what that data is. I looked at translate/translate@65b4000, and I could see that some of the languages are missing "other". All languages must have at least "other", that is a hard requirement. If a language has only one plurals form, it should be "other". I would include a CI test case that checks that all languages have "other" in them. I can see that be, pl, and uk are missing "other".
No, not all languages use other for integers (that's what android cares for plurals), see for example Belarusian, Polish or Russian.
And yes translate/translate@65b4000 is think I'd like to get reviewed (or more precisly https://github.com/translate/translate/pull/3762/files).
I agree with @eighthave. Android requires "other" and if it does not exist this can cause a crash.
So far I thought that Android is simply following CLDR, but that doesn't seem to be the case, so this has to be implemented somewhere (apparently documentation doesn't seem to exist).
To me it really looks weird that it would require the "other" even for languages where it won't be used - the getQuantityString used to evaluate this accepts only integers...
Looking at the code I still think that "other" is not needed as long as all valid choices are there.
So can somebody actually test that other is really needed in all cases? Testing simple string with one of above mentioned languages (eg. Polish) should do that.
I've test it and can confirm, some languages don't require the other field, here's some tests that I did:
The test consists on loop through 0 to 220 and check if the the getQuantityString returns the language code or throws an error (using "language_code" + %1$din the strings).
| API | 25 | 23 | 19 | 16 |
|---|---|---|---|---|
| Without field "other" | only be, pl, ru, uk |
same as 25 | same as 25, but some fallback | same as 19 |
| Without field "other" (plural deleted if empty) |
same as above, some fallback |
same as 25 | same as 25 | same as 25 |
| Following PR rules strictly | ✓ | ✓ | ✓, except bem, bez |
same as 19 |
- Exception when fail:
Resources$NotFoundException: Plural resource ID #id quantity=x item=other - You cannot have an empty plurals tag
- Deleted plurals just fallback to default
You can find the generator (to generate the strings.xml file and their contents) that follows the rules of PLURAL_TAGS that it's in your pull request (translate/translate#3762) and the app source code here:
https://github.com/mauriciocolli/LanguageTest
There's some already compiled apks if you want to test in your device.
@mauriciocolli Thanks for verifying this! I've just deployed this change to Hosted Weblate, so that will give these changes some more real life testing.
There are still few languages whose plural rules in CLDR differ to what is usually used in Gettext, so these will need additional fixes on the Weblate side:
- Lithuanian (see also #901)
- Cornish
- Upper Sorbian
- Hebrew (see #1781 (comment))
- Lower Sorbian
Thank you for your report, the issue you have reported has just been fixed in Weblate. The complete fix depends on translate/translate#3762 (which is already deployed in Hosted Weblate).
- In case you see problem with the fix, please comment on this issue.
- In case you see similar problem, please open separate issue.
- If you are happy with the outcome, consider supporting Weblate by donating.