sphinx-doc/sphinx-intl

i18n: Translated msgid strings in content files not merged into HTML output

sfc-gh-mpeters opened this issue · 21 comments

sfc-gh-mpeters commented

Since upgrading from Sphinx 1.8 to 2.1, the translated strings in the PO files in my /locale//LC_MESSAGES/ directory are no longer merged into the HTML output. However, the translated strings for my RTD theme are merged into the output. I've upgraded my copy of sphinx-intl to v2.0.0 in case that was the issue; it isn't.

I'm following the same process as before the Sphinx upgrade:

  1. Run make gettext to produce a set of POT files.
  2. Run sphinx-intl update -p build/locale -l de (to produce PO files for German translation).
  3. Translate the strings. For example:

#: ../../source/index.rst:8
msgid "English string"
msgstr "German string"

  1. Run the command to build my HTML output:

make -e SPHINXOPTS="-D language='de' -D locale_dirs='locale/,_themes/rtd_theme/locale/' -D html_theme='rtd_theme'" BUILDDIR="build/de" clean html

Sphinx creates the MO files from my PO files. The HTML output shows the translated strings for my theme but not for my content files.

When I downgrade my environment and use Python 2.7 tools to build translated output (into German), Sphinx successfully replaces the strings in my RST files with the corresponding strings in my PO files in the HTML output:
i18n-translated-strings-not-merged-into-output.zip

Python 2.7
Sphinx 1.8.4
Sphinx-intl 0.9.9

However, when I switch back to the Python 3 tools, Sphinx does replace the strings in my RTD theme sphinx.po file but does not replace the strings in my RST content files (index.po, .po):

Python 3.5
Sphinx 2.2.0
Sphinx-intl 2.0.0

I don't see any earlier reports of similar issues. Any idea what could be going on? The msgid strings in my PO files line up with the strings in my RST files, so that isn't the issue.

Mark

tk0miya commented

It seems default value of gettext_compact has been changed unexpectedly. As a workaround, setting gettext_compact = False forcedly in conf.py on my local. I'll to fix this later.

tk0miya commented

Wait, my comment above was wrong. It seems both v1.8 and v2.2 are gettext_compact = True.

Case v2.2.1

$ unzip i18n-translated-strings-not-merged-into-output.zip
$ cd source
$ touch conf.py
$ sphinx-build --version
sphinx-build 2.2.1
$ sphinx-build . _build/locale -b gettext -d _build/doctrees
(snip)
$ find _build/locale
_build/locale
_build/locale/sql-reference.pot
_build/locale/index.pot

Case 1.8.6

$ sphinx-build --version
sphinx-build 1.8.6+
$ sphinx-build . _build/locale -b gettext -d _build/doctrees -D master_doc=index
(snip)
$ find _build/locale
_build/locale
_build/locale/sql-reference.pot
_build/locale/index.pot

On the other hand, your attached project contains a message catalog file generated by gettext_compact = False option. It seems mismatched. I think this is a reason of this issue.

$ find locale
locale
locale/de
locale/de/LC_MESSAGES
locale/de/LC_MESSAGES/sql-reference
locale/de/LC_MESSAGES/sql-reference/conventions.po
locale/de/LC_MESSAGES/sql-reference/conventions.mo

Which setting do you want to use?

sfc-gh-mpeters commented
tk0miya commented

Sphinx expects gettext_compact setting is same between a project generating POT file and a project generating translated documents.

It seems first one is configured as gettext_compact = False because POT files are generated per document. On the other hand, second one is not configured. So it is considered as gettext_compact = True because of default setting. I think to append gettext_compact = False to your conf.py will resolve your problem. Could you try it please?

sfc-gh-mpeters commented
tk0miya commented

Sorry for confusing. I meant "first one" is "a project generating POT file ". And "the second one" is "a project generating translated documents".
I succeed to generate translated document with adding gettext_compact = False to attached project with Sphinx-2.2. Could you check your conf.py please? if gettext_compact is already disabled, I need more information for this trouble in detail.

sfc-gh-mpeters commented
sfc-gh-mpeters commented

@tk0miya ^^

(Sorry for the bump. I'm hoping to resolve this issue soon, because Python 2.7 is now out of date.)

Thanks.

tk0miya commented

Sorry for late.
Please upload (example) project again and let me know your procedure step by step. As I commented last, I could succeed to build translated HTML on my local. So I need to know what happened in your computer.

sfc-gh-mpeters commented

Thanks @tk0miya, and NP!

Attached is a small sample project. This is my first time assembling a project to share. Please let me know if I can provide additional files or information.

I can build translated (Japanese) documentation by executing the following command. The command uses my local Python 2.7 executable, as well as the Python 2 version of the sphinx-build executable:

make -f Makefile2 -e PYTHON=python2.7 SPHINX-BUILD=/Users/mpeters/Library/Python/2.7/bin/sphinx-build SPHINXOPTS="-D language='ja' -D locale_dirs='locale/,_themes/snowflake_rtd_theme/locale/'" BUILDDIR="build/ja/python2.7" clean html

Note that calling Makefile2 is required by my team because the version of Makefile in our source control system is configured for Engineering builds. The SPHINXBUILD and other settings used by the Doc team members are different.

When I execute the same command as above but use Python 3 and the Python 3 version of the sphinx-build executable, the strings in the build are not translated:

Makefile2 -e PYTHON=python3 SPHINX-BUILD=/usr/local/bin/sphinx-build SPHINXOPTS="-D language='ja' -D locale_dirs='locale/,_themes/snowflake_rtd_theme/locale/'" BUILDDIR="build/ja/python3" html

Both builds are included in my attached sample project, under sphinx/build/ja/.

Thanks in advance for any help you can provide!
Mark

sphinx-intl-issue-35-markpeters-snowflake.zip

tk0miya commented

Strange. On my local, both are succeeded to translate. This is Dockerfile I used to reproduce.

FROM python:3.8-slim
#FROM python:2.7-slim

RUN pip install Sphinx
RUN apt update; apt install -y curl less make unzip
RUN curl -LO https://github.com/sphinx-doc/sphinx-intl/files/4140815/sphinx-intl-issue-35-markpeters-snowflake.zip
RUN unzip sphinx-intl-issue-35-markpeters-snowflake.zip
WORKDIR /sphinx
RUN rm -rf build
RUN mkdir -p /GSCommon/src/main/resources/net/snowflake/common
RUN echo 'version: 1.0' > /GSCommon/VERSION
RUN touch /GSCommon/src/main/resources/net/snowflake/common/version.properties
RUN mkdir -p /Client
RUN touch /Client/pom.xml
RUN make -f Makefile2 -e PYTHON=python3.8 SPHINX-BUILD=sphinx-build SPHINXOPTS="-D language='ja' -D locale_dirs='locale/,_themes/snowflake_rtd_theme/locale/'" BUILDDIR="build/ja/python3.8" clean html
#RUN make -f Makefile2 -e PYTHON=python2.7 SPHINX-BUILD=sphinx-build SPHINXOPTS="-D language='ja' -D locale_dirs='locale/,_themes/snowflake_rtd_theme/locale/'" BUILDDIR="build/ja/python2.7" clean html

In addition, the structure of new example is flat. There are no subdirectories on source directory. So your trouble is not related with gettext_compact.

Anyway, I suppose this trouble would come from environment setting. Could you paste a building log of Sphinx from your console please? It might contains some hints.

sfc-gh-mpeters commented

@tk0miya Thanks so much for your quick reply!

Could you paste a building log of Sphinx from your console please? It might contains some hints.

Here you go:

$ make -f Makefile2 -e PYTHON=python3 SPHINX-BUILD=/usr/local/bin/sphinx-build SPHINXOPTS="-D language='ja' -D locale_dirs='locale/,_themes/snowflake_rtd_theme/locale/'" BUILDDIR="build/ja/python3" html
python3 /Users/mpeters/Library/Python/2.7/bin/sphinx-build -b html -d build/ja/python3/doctrees -D extensions='sqlsyntax-lexer','sqlexample-lexer','sqlexample-lower-lexer','sqljson-lexer','sphinx.ext.todo','sphinx.ext.ifconfig','sphinx_sitemap' source build/ja/python3/html
Running Sphinx v2.2.0
loading translations [en]... done
making output directory... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 5 source files that are out of date
updating environment: [new config] 5 added, 0 changed, 0 removed
reading sources... [100%] user-guide-intro                                                                                                                                                                
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] user-guide-intro                                                                                                                                                                 
generating indices...  genindexdone
writing additional pages...  searchdone
copying images... [100%] images/sf-hero.jpg                                                                                                                                                               
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

The HTML pages are in build/ja/python3/html.
sphinx-sitemap error: neither html_baseurl nor site_url are set in conf.py. Sitemap not built.

Build finished. The HTML pages are in build/ja/python3/html.

Thanks,
Mark

tk0miya commented

It seems Sphinx does not recognize giving option via -D option. It tries to load English translations.

loading translations [en]... done

Could you check your real Makefile please? I guess it forgets to pass $SPHINXOPTS to sphinx-build.

sfc-gh-mpeters commented

@tk0miya

Could you check your real Makefile please? I guess it forgets to pass $SPHINXOPTS to sphinx-build.

Do you just mean, make sure the html settings in Makefile include $(SPHINXOPTS) or $(ALLSPHINXOPTS)? If so, then yes, they do.

html:
	$(PYTHON) $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
	@echo
	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

Thanks,
Mark

sfc-gh-mpeters commented

I tried modifying my local copy of Makefile (avoiding Makefile2) as follows:

Change the SPHINXBUILD and Python the variables to point to my Python 3 version of the sphinx-build executable and python3, respectively:

SPHINXOPTS    = 
SPHINXBUILD   = /usr/local/bin/sphinx-build
PAPER         =
BUILDDIR      = build
PYTHON        = python3

Then I added a custom set of build instructions:

ja-html:
	$(PYTHON) $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) -D language='ja' $(BUILDDIR)/html
	@echo
	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

When I built the Japanese doc, the log reported that the Japanese translations were loaded:

make clean ja-html
rm -rf build/*
python3 /usr/local/bin/sphinx-build -b html -j 1 -d build/doctrees   source -D language='ja' build/html
Running Sphinx v2.2.0
loading translations [ja]... done
making output directory... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 5 source files that are out of date
updating environment: [new config] 5 added, 0 changed, 0 removed
reading sources... [100%] user-guide-intro                                                                                                                                                                
<rst_prolog>:5: WARNING: Error in "replace" directive: may contain a single paragraph only.
<rst_prolog>:5: WARNING: Substitution definition "snowsql_version" empty or invalid.

.. |snowsql_version| replace:: 1.2.2

 MIT ライセンス（MIT）
<rst_prolog>:5: WARNING: Error in "replace" directive: may contain a single paragraph only.
<rst_prolog>:5: WARNING: Substitution definition "snowsql_version" empty or invalid.

.. |snowsql_version| replace:: 1.2.2

 MIT ライセンス（MIT）
/Users/mpeters/documentation/git/branches/production_3.56_200010_sphinx/Docs/sphinx/source/sphinx_rtd_theme_license.rst:5:<translated>:2: WARNING: Explicit markup ends without a blank line; unexpected unindent.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] user-guide-intro                                                                                                                                                                 
generating indices...  genindexdone
writing additional pages...  searchdone
copying images... [100%] images/sf-hero.jpg                                                                                                                                                               
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 5 warnings.

The HTML pages are in build/html.

But the built HTML files are still in English only.

make clean ja-html
rm -rf build/*
python3 /usr/local/bin/sphinx-build -b html -j 1 -d build/doctrees   source -D language='ja' build/html
Running Sphinx v2.2.0
loading translations [ja]... done
making output directory... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 5 source files that are out of date
updating environment: [new config] 5 added, 0 changed, 0 removed
reading sources... [100%] user-guide-intro                                                                                                                                                                
<rst_prolog>:5: WARNING: Error in "replace" directive: may contain a single paragraph only.
<rst_prolog>:5: WARNING: Substitution definition "snowsql_version" empty or invalid.

.. |snowsql_version| replace:: 1.2.2

 MIT ライセンス（MIT）
<rst_prolog>:5: WARNING: Error in "replace" directive: may contain a single paragraph only.
<rst_prolog>:5: WARNING: Substitution definition "snowsql_version" empty or invalid.

.. |snowsql_version| replace:: 1.2.2

 MIT ライセンス（MIT）
/Users/mpeters/documentation/git/branches/production_3.56_200010_sphinx/Docs/sphinx/source/sphinx_rtd_theme_license.rst:5:<translated>:2: WARNING: Explicit markup ends without a blank line; unexpected unindent.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] user-guide-intro                                                                                                                                                                 
generating indices...  genindexdone
writing additional pages...  searchdone
copying images... [100%] images/sf-hero.jpg                                                                                                                                                               
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 5 warnings.

The HTML pages are in build/html.
sfc-gh-mpeters commented

@tk0miya

Sorry to keep bothering you. If you can build translated output using my own project files and Python 3 and a version of Sphinx for Python 3, then clearly there's nothing wrong with the sphinx-intl utility.

But I was wondering: Do you know what it could mean when my build log shows Sphinx is loading the translated PO files...

loading translations [ja]... done

... but the HTML files in my build directory are not translated?

Thanks again,
Mark

tk0miya commented

On using Makefilie2, you give -D locale_dirs='locale/,_themes/snowflake_rtd_theme/locale/' to sphinx. But it is not given to sphinx in your latest report.

python3 /usr/local/bin/sphinx-build -b html -j 1 -d build/doctrees   source -D language='ja' build/html

Is this okay? loading translations [ja] message only says "Trying to load Japanese translations". If you have a wrong settings, it goes bad.

Please, please, please check your configuration and environment. It seems your build environment is to o complex. I don't know why you're using -D option instead of conf.py. But I feel it makes your problem difficult. I think it is not to difficult to resolve it. Please check your command-line, configurations and logs step by step.

Thanks,

sfc-gh-mpeters commented

Thanks @tk0miya. I really appreciate your help.

I've simplified my build configuration:

Makefile

  • Modified SPHINXBUILD to point to my Python 3 version of the sphinx-build executable.
  • Modified PYTHON to read python3.

conf.py

Made the following changes:

language = 'ja'

locale_dirs = ['locale/']
gettext_compact = False

When I run make clean html, the log seems to indicate the translated strings were merged successfully. But the output HTML files are in English.

`$ make html
python3 /usr/local/bin/sphinx-build -b html -j 1 -d build/doctrees source build/html
Running Sphinx v2.2.0
loading translations [ja]... done
making output directory... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 5 source files that are out of date
updating environment: [new config] 5 added, 0 changed, 0 removed
reading sources... [100%] user-guide-intro
<rst_prolog>:5: WARNING: Error in "replace" directive: may contain a single paragraph only.
<rst_prolog>:5: WARNING: Substitution definition "snowsql_version" empty or invalid.

.. |snowsql_version| replace:: 1.2.2

MIT ライセンス（MIT）
<rst_prolog>:5: WARNING: Error in "replace" directive: may contain a single paragraph only.
<rst_prolog>:5: WARNING: Substitution definition "snowsql_version" empty or invalid.

.. |snowsql_version| replace:: 1.2.2

MIT ライセンス（MIT）
/Users/mpeters/documentation/git/branches/production_3.56_200010_sphinx/Docs/sphinx/source/sphinx_rtd_theme_license.rst:5::2: WARNING: Explicit markup ends without a blank line; unexpected unindent.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] user-guide-intro
generating indices... genindexdone
writing additional pages... searchdone
copying images... [100%] images/sf-hero.jpg
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 5 warnings.

The HTML pages are in build/html.

Build finished. The HTML pages are in build/html.`

Thanks for your help. I'll continue to try to resolve the issue; and, if I learn anything useful, will share it in this ticket. In the meantime, this ticket can be closed.

Best,
Mark

tk0miya commented

I recommend you to restart with small project or scratch. As I commented above, I succeeded to build translated document via sphinx-intl-issue-35-markpeters-snowflake.zip you attached here. Did you try it? I still don't know why you're in failure. But I guess this comes from environment issue or build script issue.
Thanks,

sfc-gh-mpeters commented

Hi @tk0miya

You're right: I was encountering a local environment issue. Our developers added the following code to our conf.py file to pull product/client version information from a properties file elsewhere in our repo. For some reason, when I remove this code from the conf.py file, translated text (e.g. Japanese) is merged successfully into my HTML builds using Sphinx 2.2/Python 3.

# The short X.Y version.
version = None
# The full version, including alpha/beta/rc tags.
release = None

version_properties = os.path.join(
    THIS_DIR, '..','..','VERSION')
lines = [line.rstrip('\n') for line in open(version_properties)]

for line in lines:
    m = re.match(r'^\s*version\s*:\s*([\d\.]+)', line)
    if m:
        release = m.group(1)
        version=release[0:release.rfind('.')]
        break

if release is None or version is None:
    print('version or release is None. Check GSCommon/VERSION includes the version number')
    sys.exit(1)

# The client version information
version_properties = os.path.join(
    THIS_DIR, '..','..','..','client','VERSION',
    'resources','net','snowflake','common','version.properties')
lines = [line.rstrip('\n') for line in open(version_properties)]

jdbc_version=None
odbc_version=None
python_connector_version=None
snowsql_version=None

version_re = re.compile(r'^.*=([\d\.]+)')
for line in lines:
    if line.startswith('odbc'):
        m = version_re.match(line)
        if m:
            odbc_version = m.group(1)
    elif line.startswith('python'):
        m = version_re.match(line)
        if m:
            python_connector_version = m.group(1)
    elif line.startswith('snowsql'):
        m = version_re.match(line)
        if m:
            snowsql_version = m.group(1)

client_pom = os.path.join(
    THIS_DIR,
    '..','..','..','Client','pom.xml')

lines = [line.rstrip('\n') for line in open(client_pom)]
for line in lines:
    m = re.match(r'^\s*<version>([\d\.]+)', line)
    if m:
        jdbc_version = m.group(1)
        break

rst_prolog = """
.. |jdbc_version| replace:: {jdbc_version}
.. |odbc_version| replace:: {odbc_version}
.. |python_connector_version| replace:: {python_connector_version}
.. |snowsql_version| replace:: {snowsql_version}
""".format(
    jdbc_version=jdbc_version,
    odbc_version=odbc_version,
    python_connector_version=python_connector_version,
    snowsql_version=snowsql_version)

I just need to figure out how to pull the information into our docs without using this code, since it seems to conflict with the merge code for translated text.

Best wishes,
Mark

tk0miya commented

Thank you for letting me know.
Unfortunately, there are nothing to do from our side. So I'm closing this now. Good luck.