[TF2] [Linux] Cannot display text outside repertoire of default font

Question

[TF2] [Linux] Cannot display text outside repertoire of default font

Opened this issue 6 months ago · 15 comments

Background

At issue #153, it was discussed that source engine games on Linux (at least TF2) would look for a font with the name "WenQuanYi Zen Hei" as discussed by this comment.

So, for the past few years, in order to see all the characters in people's usernames, I would take GNU's Unifont, and change its information so that it would be WenQuanYi Zen Hei. This has worked for the longest time and on many different huds, but it has stopped working due to this update.

My Thoughts

I believe there are two reasons as to why this may be happening. First off, I recently heard that TF2 now has emoji support on Linux, which I did see in my game. So, maybe the addition of this emoji support changed something, and now my game isn't looking for WenQuanYi Zen Hei, and I am unable to see non-ASCII fonts anymore outside of emojis. Secondly, it could be now that TF2 is running under the Sniper Steam Runtime, any fonts I have installed in my system won't be used by the game. This is unfortunate.

Why It Matters

The world is a huge place, and not everybody's name is just constructed out of ASCII characters. It's very frustrating to see a leaderboard with missing characters.

Example

I put a ඞ in my username, and here is a screenshot of my main screen.

Answer 1 · 2024-04-21T22:11:03.000Z

Hello @HurricanePootis, given the similarity to steam-runtime ValveSoftware/steam-runtime#555, I've transferred this issue report to the steam-runtime issue tracker.

Can you note exactly where the font you need is located on the host system?

Answer 2 · 2024-04-21T22:30:07.000Z

Hello @kisak-valve ,

I have the font installed at ~/.local/share/fonts/unifont-15.1.04-WenQuanYiZenHei.ttf. I also run fc-cache -fv to rebuild the cache.

Answer 3 · 2024-05-01T14:51:21.000Z

Like the other titles, we no longer use this obscure font but use Noto Sans installed on the system now.

Answer 4 · 2024-05-01T23:55:28.000Z

Like the other titles, we no longer use this obscure font but use Noto Sans installed on the system now.

I have the noto sans fonts installed on my systems, both noto-fonts and noto-fonts-cjk from the archlinux repos. I also have ttf-twemoji installed for emoji fonts, which is used by the game.

Answer 5 · 2024-05-03T18:17:05.000Z

I put a ඞ in my username

(For reference, this is U+0D9E SINHALA LETTER KANTAJA NAASIKYAYA)

Answer 6 · 2024-05-28T18:44:55.000Z

Like the other titles, we no longer use this obscure font but use Noto Sans installed on the system now.

If TF2 is no longer asking the OS for a font named WenQuanYi Zen Hei, then it is expected that a modified font of that name will not be read. It will load whatever font name(s) the game developer has told it to read, and hopefully fall back to using any available font that is found by the fontconfig library and has the appropriate glyphs.

There are three sources of fonts that can potentially work for TF2 and other games that run in the SteamLinuxRuntime_sniper container:

The game can bundle fonts that include suitable glyphs for all languages that the game developer aims to support, in the visual style they prefer. The English text in your screenshot is presumably using a font included with the game, but U+0D9E SINHALA LETTER KANTAJA NAASIKYAYA is presumably not included in that font.
There is code derived from Flatpak to take your fonts from ~/.local/share/fonts, /usr/local/share/fonts and /usr/share/fonts on the host system, and make them available to the container: ValveSoftware/steam-runtime#555 (comment). In this case, it is probably this mechanism that you are hoping will be used to load a fallback font provided by your operating system.
The runtime has a copy of Deja Vu, which is used as a fallback/font of last resort. I don't think this contains east Asian glyphs, in an effort to limit the size of the runtime: if we included all of Noto, that would cost about 800MB (plus another 800MB for the soldier runtime, and the same again for each future runtime). So this is unlikely to be able to supply U+0D9E SINHALA LETTER KANTAJA NAASIKYAYA.

If you are hoping for your OS-wide fonts to be used as a fallback, then a log file from launching and exiting the game with STEAM_LINUX_RUNTIME_VERBOSE=1 in the environment would be useful: please see https://github.com/ValveSoftware/steam-runtime/blob/master/doc/reporting-steamlinuxruntime-bugs.md#essential-information. That will tell us more about how the container runtime framework maps your OS-wide fonts into the container.

It would also be useful to confirm which of your installed fonts (if any) has U+0D9E SINHALA LETTER KANTAJA NAASIKYAYA in its repertoire of glyphs. I would expect that it's probably in Noto Sans Sinhala, which is not part of CJK; in Debian it's packaged as part of fonts-noto-core, but the way it's packaged in Arch might well be different.

Answer 7 · 2024-06-02T03:37:49.000Z

@smcv Here are my findings:

I'm using Fedora Linux 40 KDE. Steam is from rpmfusion-nonfree; I'm not using the Flatpak version.

List of installed fonts on my system: https://0.jaegers.net/?02b0401f0fbdf616#D4GtQHSkiGQSA2FsgykR3Pt9XJjKddUDDPEqioJRxRAe

Output of fc-list ":charset=0x0D9E" (as I understand it, this should print every font that supports U+0D9E SINHALA LETTER KANTAJA NAASIKYAYA): https://0.jaegers.net/?2e34fbb69f387b43#AAwAJhZEwg7Xku8GjtVe13oq2X6Ri5aoGKd2bTexuBAe

My ~/.config/fontconfig/fonts.conf file: https://0.jaegers.net/?f8cf07780cd74506#2gKsCVErKhnyK2C1qbDjkoR7DUT8Q5L6fJGFR2N6kAPD

Output from Steam Client's "Help -> Steam Runtime Diagnostics": https://0.jaegers.net/?225ccb328e340a0b#4EXgmt6eVtrq2un5N3izRztUBqpduPB9cLr3tdEdrrmP

Steam Linux Runtime Sniper log output (slr-app440-t20240602T002224.log): https://0.jaegers.net/?e913b15f340a1366#GxLBMFu678WhQEabsAfQ4rU2TqrNMin2UVHkDDx4zb4B

Content of ~/.steam/root/ubuntu12_32/steam-runtime/version.txt:

steam-runtime_0.20240415.84615

Content of steamapps/common/SteamLinuxRuntime_sniper/VERSIONS.txt:

#Name	Version		Runtime	Runtime_Version	Comment
depot	0.20240415.84603			# Overall version number
pressure-vessel	0.20240415.0	scout		# pressure-vessel-bin.tar.gz
scripts	0.20240415.0			# from steam-runtime-tools
sniper	0.20240415.84603	sniper	0.20240415.84603	# sniper_platform_0.20240415.84603/

Screenshot of how

ඞ
€, š, đ, č, ć, ž
( ͡° ͜ʖ ͡°)

render in the in-game chat and scoreboard, respectively:

Answer 8 · 2024-06-03T18:25:03.000Z

I have updated my fonts.conf file to include "symbols" fonts, but the problem persists. The game doesn't seem to care about system font configuration.

https://0.jaegers.net/?111e5dbd07dcfbad#49yqWKaGwSXrhCan716AooqdWaumPdXUs9EE4es6NYXy

Answer 9 · 2024-06-07T17:06:45.000Z

# ==============
# System default
# ==============

This is not valid: comments in XML look . I don't know whether fontconfig silently ignores arbitrary text in unexpected places; but if it doesn't, then that might be contributing to this.

Valid multi-line comments could be something like

<!--
=============
System default
=============
-->

but make sure the text of the comment doesn't contain --.

(XML is weird like that, and it would probably have been better if fontconfig had chosen something more human-editable...)

Answer 10 · 2024-06-07T17:08:51.000Z

The game doesn't seem to care about system font configuration

Part of the purpose of the container runtime mechanism is to insulate the game from implementation details of the host system, so that games that work on Ubuntu will also work on (for example) Arch, Debian and Fedora. So it is not unexpected that some details of what happens on the host system are ignored by the container: for instance, we specifically don't read /etc/fonts/ because if it was designed for a different version of fontconfig, it might not be compatible with what happens outside the container.

I think ~/.config/fonts/ does get into the container, though.

Answer 11 · 2024-06-07T17:23:48.000Z

I have updated my fonts.conf file to include "symbols" fonts, but the problem persists.

I would not expect adding Noto Sans Symbols to the list to have any impact on whether/how U+0D9E is displayed, because you said in #6058 that it isn't one of your fonts that contains U+0D9E. I would expect a "Symbols" font to contain e.g. mathematical symbols, but none of the glyphs for human languages like Sinhala.

If you add Noto Sans Sinhala to the preference list, does that change anything?

Answer 12 · 2024-06-07T17:59:46.000Z

@smcv Thanks for noticing the comment format. I have corrected them.

To be clear, I don't have the directory ~/.config/fonts/ as you mentioned, but I do have ~/.config/fontconfig/ (I think this is what you meant).

My new ~/.config/fontconfig/fonts.conf file: https://0.jaegers.net/?8fcc385e30d32952#Ch92PjBo8uidVCVAHEbVptcT3vULuhPBd2agGbpR2Gfq

What I did: updated the fonts.conf file (adjusted comments and added Noto Sans Sinhala and Noto Serif Sinhala), ran fc-cache-32 -f -r && fc-cache-64 -f -r && sudo fc-cache-32 -f -r && sudo fc-cache-64 -f -r in the terminal, exited Steam completely, then opened Steam again and opened TF2 to test the changes.

In-game chat and scoreboard screenshots after the changes:

Same results as before.

If you add Noto Sans Sinhala to the preference list, does that change anything?

I think it would not make any difference (as it didn't), because in my previous test I had Unifont and Last Resort in the preference list and both of them support U+0D9E SINHALA LETTER KANTAJA NAASIKYAYA.

Answer 13 · 2024-06-07T19:02:32.000Z

On closer inspection I think this is a limitation of how TF2's engine or graphics subsystem handles fonts, rather than a problem with the SLR container, so it should be retitled to something like "[TF2] [Linux] Cannot display text outside repertoire of default font" and moved back to the TF2 issue tracker.

@kisak-valve, if you agree with my conclusion, perhaps you could do that?

(I assume that Sinhala was just an arbitrary example, and there are probably other scripts that are equally affected: I would guess that everything that has its own separate font in the Noto family, such as Arabic, Hebrew and Thai, is probably in the same situation.)

I don't know precisely what TF2 is doing to select which font it will use to render each character, but the tl;dr of the debugging below is that GTK can render Sinhala text successfully, but a naive use of fontconfig and SDL2_ttf cannot. This is equally true either inside or outside the container, which I think implies that the container is not what is breaking this.

TF2 is not open-source, but I suspect that its font rendering may have more in common with my example of a naive use of fontconfig and SDL2_ttf than it does with GTK.

Debugging

On an Ubuntu 24.04 system with fonts-noto-core installed, if I run the GTK program zenity and tell it to display Sinhala:

zenity --info --text="$(cat ~/tmp/sinhala.txt)"

(where ~/tmp/sinhala.txt contains U+0D9E, which can be typed as Ctrl+U 0 D 9 E Space in GNOME), I get what appears to be a correct glyph displayed. I can't read Sinhala myself, but the glyph looks something like the one in the Unicode Consortium's reference chart so I assume it's right.

But if I run this, which uses fontconfig to load "sans-serif" into a SDL2_ttf font object and then uses that font object to render the text:

~/.steam/root/ubuntu12_32/steam-runtime/amd64/usr/bin/steam-runtime-dialog-ui --info --text="$(cat ~/tmp/sinhala.txt)"

(exactly the same options, with U+0D9E between the quotes), I just get a placeholder rectangle for the Sinhala glyph.

I think what is happening here is that when GTK renders text (via Pango), it chooses fonts individually for each glyph that it displays, starting with the requested font if it can but falling back to other fonts if necessary. But when steam-runtime-dialog-ui renders text, it starts by loading a single font, and then renders the whole string via that font. The default sans-serif font is generally going to be something like DejaVu or Noto Sans, so glyphs that are present in the selected font (such as Latin, Cyrillic, Greek) will be displayed successfully; but if any glyph is missing from the selected font, it will be replaced by the placeholder (even if another font might actually have been able to provide it).

For comparison, if I set TF2's launch options to PRESSURE_VESSEL_SHELL=instead %command%, I get an interactive shell inside the container. Inside that shell, I can do:

unset LC_ALL       # bypassing LC_ALL=C so we can handle UTF-8 again
zenity --info --text="$(cat ~/tmp/sinhala.txt)"
steam-runtime-dialog-ui --info --text="$(cat ~/tmp/sinhala.txt)"

and the results are the same as outside the container: zenity (GTK) displays the Sinhala glyph, but steam-runtime-dialog-ui (naive use of fontconfig and SDL2_ttf) does not.

A possible solution might be to render text using something like Pango instead of SDL_ttf - which is a much more complicated library, but it solves a much more complicated problem.

Answer 14 · 2024-06-10T11:10:32.000Z