untra/polyglot

site.active_lang is occasionally wrong when parallel_localization is enabled

linsui opened this issue · 8 comments

We upgraded Jekyll from 3.8.3 to 4.3.1 for https://f-droid.org/ and had to upgrade polyglot to get that working. Since then, site.active_lang is occasionally the wrong value, causing parts of the page to be rendered in the wrong language. It seems to always be a language that was processed as part of the same batch. For example, on our 12 CPU render machine, de occasionally gets he and zh_Hans occasionally gets tr.

It seems that setting parallel_localization: false fixes it, but also drastically slows things down for a 24 language site.

@obfusk Could you also take a look since you worked on parallel_localization? Thanks!

With parallel_localization: false, there is now a stacktrace involving polyglot:
https://gitlab.com/fdroid/fdroid-website/-/jobs/6364604629

I am no longer working on anything related to F-Droid, having been given no choice but to leave the project several months ago.

图片
图片

could be related to this prior change. But writing tests for some of this is rather challenging. The parallel localization approach to fork is so naive, and I regret it as a design decision from a security perspective.
https://github.com/untra/polyglot/pull/140/files

I would appreciate help with the current open PRs @obfusk started, but I also understand if ya also want nothing to do with this. I could use the help with open source software tho.

@linsui did you upgrade any other parts of the site build process besides jekyll major versions by chance? different ci provider, new arm runners by chance? those can influence weird behavior problems like this given the fork strategy parallel_localization uses.

Hey @untra thanks for your response. I've been working with @linsui to troubleshoot this, but I'm not a Ruby or Jekyll programmer. Switching to parallel_localization: false in https://gitlab.com/fdroid/fdroid-website/-/jobs/6364604629 actually gave us a really different result, so perhaps this is not a bug in the parallelization after all. There is a stacktrace now, and I am guessing it might have something to do with changes made to how default_lang: is handled since v1.3ish. Here's the stacktrace:

  Liquid Exception: Liquid error (/builds/fdroid/fdroid-website/_includes/sidebar-lastupdated-packages.html line 3): Cannot sort a null object. included in /builds/fdroid/fdroid-website/_layouts/default.html
/usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/filters.rb:302:in `sort': Liquid error (/builds/fdroid/fdroid-website/_includes/sidebar-lastupdated-packages.html line 3): Cannot sort a null object. included  (Liquid::ArgumentError)
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/strainer_template.rb:52:in `invoke'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/context.rb:104:in `invoke'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/variable.rb:90:in `block in render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/variable.rb:88:in `each'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/variable.rb:88:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/tags/assign.rb:37:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:80:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:230:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:213:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/document.rb:41:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:194:in `render'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-fdroid-aafd701ad4df/lib/jekyll/FDroidLastUpdatedPackagesTag.rb:26:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/tag.rb:51:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:80:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:230:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:213:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/document.rb:41:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:194:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:204:in `render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:39:in `block (3 levels) in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:59:in `measure_counts'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:38:in `block (2 levels) in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:63:in `measure_bytes'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:37:in `block in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:70:in `measure_time'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:36:in `render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/tags/include.rb:122:in `block in render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/context.rb:134:in `stack'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/tags/include.rb:119:in `render'
	from /usr/share/rubygems-integration/all/gems/jekyll-include-cache-0.2.1/lib/jekyll-include-cache/tag.rb:22:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/tag.rb:51:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:80:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:230:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:213:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/document.rb:41:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:194:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:204:in `render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:39:in `block (3 levels) in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:59:in `measure_counts'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:38:in `block (2 levels) in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:63:in `measure_bytes'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:37:in `block in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:70:in `measure_time'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:36:in `render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:129:in `render_liquid'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:192:in `render_layout'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:161:in `place_in_layouts'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:93:in `render_document'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:63:in `run'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:578:in `render_regenerated'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:563:in `block (2 levels) in render_docs'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:562:in `each'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:562:in `block in render_docs'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:561:in `each_value'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:561:in `render_docs'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:210:in `render'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:80:in `process'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:101:in `process_active_language'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:85:in `process_language'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:60:in `block in process'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:59:in `each'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:59:in `process'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/command.rb:28:in `process_site'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/commands/build.rb:65:in `build'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/commands/build.rb:36:in `process'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/command.rb:91:in `block in process_with_graceful_fail'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/command.rb:91:in `each'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/command.rb:91:in `process_with_graceful_fail'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `block in execute'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `each'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `execute'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary/program.rb:44:in `go'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/exe/jekyll:13:in `<top (required)>'
	from /usr/bin/jekyll:25:in `load'
	from /usr/bin/jekyll:25:in `<main>'
/usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/filters.rb:302:in `sort': Cannot sort a null object. (ArgumentError)
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/strainer_template.rb:52:in `invoke'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/context.rb:104:in `invoke'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/variable.rb:90:in `block in render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/variable.rb:88:in `each'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/variable.rb:88:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/tags/assign.rb:37:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:80:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:230:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:213:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/document.rb:41:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:194:in `render'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-fdroid-aafd701ad4df/lib/jekyll/FDroidLastUpdatedPackagesTag.rb:26:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/tag.rb:51:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:80:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:230:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:213:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/document.rb:41:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:194:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:204:in `render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:39:in `block (3 levels) in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:59:in `measure_counts'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:38:in `block (2 levels) in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:63:in `measure_bytes'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:37:in `block in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:70:in `measure_time'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:36:in `render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/tags/include.rb:122:in `block in render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/context.rb:134:in `stack'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/tags/include.rb:119:in `render'
	from /usr/share/rubygems-integration/all/gems/jekyll-include-cache-0.2.1/lib/jekyll-include-cache/tag.rb:22:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/tag.rb:51:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:80:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:230:in `render_node'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/block_body.rb:213:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/document.rb:41:in `render_to_output_buffer'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:194:in `render'
	from /usr/share/rubygems-integration/all/gems/liquid-5.4.0/lib/liquid/template.rb:204:in `render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:39:in `block (3 levels) in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:59:in `measure_counts'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:38:in `block (2 levels) in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:63:in `measure_bytes'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:37:in `block in render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:70:in `measure_time'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/liquid_renderer/file.rb:36:in `render!'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:129:in `render_liquid'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:192:in `render_layout'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:161:in `place_in_layouts'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:93:in `render_document'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/renderer.rb:63:in `run'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:578:in `render_regenerated'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:563:in `block (2 levels) in render_docs'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:562:in `each'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:562:in `block in render_docs'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:561:in `each_value'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:561:in `render_docs'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:210:in `render'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/site.rb:80:in `process'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:101:in `process_active_language'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:85:in `process_language'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:60:in `block in process'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:59:in `each'
	from /var/lib/gems/3.1.0/bundler/gems/jekyll-polyglot-0760eda11055/lib/jekyll/polyglot/patches/jekyll/site.rb:59:in `process'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/command.rb:28:in `process_site'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/commands/build.rb:65:in `build'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/commands/build.rb:36:in `process'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/command.rb:91:in `block in process_with_graceful_fail'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/command.rb:91:in `each'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/command.rb:91:in `process_with_graceful_fail'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `block in execute'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `each'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `execute'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary/program.rb:44:in `go'
	from /usr/share/rubygems-integration/all/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
	from /usr/share/rubygems-integration/all/gems/jekyll-4.3.1/exe/jekyll:13:in `<top (required)>'
	from /usr/bin/jekyll:25:in `load'
	from /usr/bin/jekyll:25:in `<main>'

try out v1.8.0, which should additionally provide further debugging output if sub processes are individually failing.

We've tried it on staging.f-droid.org and it seems like it might reduce the likelihood of the sidebar element having the wrong language, but it doesn't completely stop it. I'm switching production to v1.8.0 and I uploaded it to Debian.