[BUG] Custom transliterations aren't take into account (`ASCII-8BIT` encoded keys?)
beauraF opened this issue · 3 comments
Hello 👋
We are using custom rules for transliteration. Since #587 they are not taking into account anymore.
It seems that this is due to the fact that symbolize_names returns keys with ASCII-8BIT encoding :
> YAML.unsafe_load_file('config/locales/en.yml', symbolize_names: false, freeze: true)
=> {"en"=>{"i18n"=>{"transliterate"=>{"rule"=>{"ö"=>"oe"}}}}}
> YAML.unsafe_load_file('config/locales/en.yml', symbolize_names: true, freeze: true)
=> {:en=>{:i18n=>{:transliterate=>{:rule=>{:"\xC3\xB6"=>"oe"}}}}}This is what @approximations contains in our HashTransliterator:
{"À"=>"A", ..., "ö"=>"o", ..., "\xC3\xB6"=>"oe" }What I tried to do
en:
i18n:
transliterate:
rule:
ö: oe > I18n.transliterate('ö', locale: :en)What I expected to happen
> I18n.transliterate('ö', locale: :en)
=> "oe"What actually happened
> I18n.transliterate('ö', locale: :en)
=> "o"Versions of i18n, rails, and anything else you think is necessary
- Rails 7.0.2.2
- Ruby 3.0.3
- I18n 1.9.1
Hi!
Are you by any chance using bootsnap? I have a very similar issue, albeit without using custom transliterations (only rails-i18n). It seems updating i18n from 1.8.11 to 1.9.1 while using bootsnap broke something:
assert_equal({ :ä => "ae", :é => "e", :ü => "ue", :ö => "oe", :Ä => "Ae", :Ü => "Ue", :Ö => "Oe", :ß => "ss" }, I18n.t("i18n.transliterate.rule", locale: :de))OK for 1.8.11 without bootsnap
OK for 1.8.11 with bootsnap
OK for 1.9.1 without bootsnap
Breaks for 1.9.1 with bootsnap:
Failure:
ApplicationHelperTest#test_all_countries_:de [/private/tmp/transliterate-bug/test/helpers/application_helper_test.rb:5]:
--- expected
+++ actual
@@ -1 +1 @@
-{:ä=>"ae", :é=>"e", :ü=>"ue", :ö=>"oe", :Ä=>"Ae", :Ü=>"Ue", :Ö=>"Oe", :ß=>"ss"}
+{:"\xC3\xA4"=>"ae", :"\xC3\xA9"=>"e", :"\xC3\xBC"=>"ue", :"\xC3\xB6"=>"oe", :"\xC3\x84"=>"Ae", :"\xC3\x9C"=>"Ue", :"\xC3\x96"=>"Oe", :"\xC3\x9F"=>"ss"}
I made a demo repository with a rails new --minimal, a single test file and branches for each combination of i18n version and bootsnap enabled/disabled:
https://github.com/lorenzk/i18n-transliterate-bug/pulls
Versions of i18n, rails, and anything else you think is necessary
- Rails 7.0.2.2
- Ruby 3.1.0
- I18n 1.9.1 and 1.8.11