Override paths: DIR_EXE = DIR_MODULE = <libcef.so.path> on Linux
Closed this issue · 41 comments
Original report by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
There is a problem with loading "icudtl.dat" and "natives_blob.bin" on Linux when executable is in a different directory than "libcef.so". In previous versions of CEF (branch 1650 for sure) these ICU/V8 files were looked for in the directory where libcef.so resided. In newer branches the code in Chromium changed and now it looks for these files in the dir where executable resides - using DIR_EXE path service key. This is a serious issue eg. in CEF Python because the python executable will almost always be in /usr/bin.
This issue was described in greater details on the CEF Forum [1] and in the CEF Python Issue Tracker [2]. I've already created a patch to expose CefOverridePath function to override arbitrary paths [3], but Marshall suggests it shouldn't be allowed to do this, as this is sensitive task and many things could go wrong if timing is wrong. Overriding DIR_EXE = DIR_MODULE = <libcef.so path> fixes the issue completely, so we should only do this.
Paths should be overriden during a call to CefExecuteProcess [libcef/context/browser.cc], as one of the first things this function does. In CEF Python I've overriden paths before calling CefExecuteProcess and only in the browser process, and that was enough to fix the issue. But now that I think of it again, paths should be overriden for all the CEF processes.
I will work on this patch soon, when I find time.
[1] Related topic on the CEF Forum: https://www.magpcss.org/ceforum/viewtopic.php?f=6&t=14271
[2] Issue explained in greater details: cztomczak/cefpython#231 (comment)
[3] Original CefOverridePath patch: https://github.com/cztomczak/cefpython/blob/0cc124e24a4852c4e74136d4413654dd2a7d7b44/patches/issue231.patch
We should only need to override DIR_EXE on Linux. DIR_MODULE should already be correct. On Windows Chromium looks for icudtl.dat in DIR_MODULE [1], and on OS X it looks in the app bundle.
The override will need to be set in CefContext::Initialize before the call to |main_runner_->Initialize|. We may also need to set it from CefExecuteProcess for sub-processes, but I'm not sure if that's necessary.
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
My testing environment is CEF Python which uses a separate subprocess executable which lies in the same directory as libcef.so. It makes me wonder whether overriding paths would affect other scenarios eg. when both the browser process and subprocess code paths are in the same executable. What if app.exe is placed in "/opt/app/" and lib/resources are placed in "/usr/local/lib/app/" and application expects that the debug.log file will be created in "/opt/app/"? If we override DIR_EXE and DIR_MODULE then this will affect placement of the log file. In main_delegate.cc there is function GetDefaultLogFile() which uses DIR_EXE. There may be more issues that I can't think of now. Maybe overriding these paths should be an option. Maybe add an additional bool argument to CefExecuteProcess that when set True will override these paths?
The user can already override the log file location via CefSettings.log_file. I think it's OK to expect resource files to be in the same directory as libcef.so/dll on Linux and Windows. Temporary files like logs should probably be created in the current working directory or a known temp path (like /var/log) instead of the DIR_EXE/DIR_MODULE location, but that's a different issue.
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
Looks like "natives_blob.bin" is also loaded from different directories on Win/Mac [1] (DIR_MODULE and base::mac::PathForFrameworkBundleResource), so overriding DIR_EXE on Linux should be enough as you say.
[1] https://cs.chromium.org/chromium/src/gin/v8_initializer.cc?q=v8_initializer&sq=package:chromium&l=83
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
Marshall wrote:
We should only need to override DIR_EXE on Linux. DIR_MODULE should already be correct.
From my testing DIR_MODULE is also wrong, so we should override both:
#!text
[CEF Python] ---- PK_DIR_EXE: /usr/bin
[CEF Python] ---- PK_DIR_MODULE: /usr/bin
[CEF Python] ---- PK_DIR_CURRENT: /home/czarek/github/cefpython/src/linux/binaries_64bit
[CEF Python] ---- PK_FILE_EXE: /usr/bin/python2.7
[CEF Python] ---- PK_FILE_MODULE: /usr/bin/python2.7
From my testing DIR_MODULE is also wrong, so we should override both:
OK, sounds good.
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
Created pull request #66: https://bitbucket.org/chromiumembedded/cef/pull-requests/66/override-paths-dir_exe-dir_module-on-linux/diff
Issue #668 was marked as a duplicate of this issue.
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
Marshall, Let's take again look at this issue. In my original patch that exposed CefOverridePath, you were worried that overriding paths is time-sensitive, so we shouldn't allow user to do that. You suggested that instead of making it configurable we should probably just set DIR_EXE = DIR_MODULE = <libcef.so path> on Linux. That is what I did in PR #66. However this caused issues with ceftests when running from the chromium/out/ directory. So you suggested that we expose a CefSettings option to set the path. I can make it that way, however I think exposing CefOverridePath was a simple and future-proof solution, because you don't know if upstream Chromium will make some changes again that will break things and another fix will be required. I don't see no advantage of having an another option like CefSettings.override_dir_exe_module_paths over making a call to CefOverridePath. In both cases you have to provide a path and need to know that this is a fix for a bug on Linux.
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
Correct pull request #66 link: https://bitbucket.org/chromiumembedded/cef/pull-requests/66/override-paths-dir_exe-dir_module-on-linux/diff
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
I know that recently there were changes to unit tests, ceftests are now build the same as cefclient and other test apps. So is overriding DIR_EXE and DIR_MODULE still an issue for tests? If not then pull request #66 can be accepted.
@czarek: Do you have a PR for the CefOverridePath change that you reference? Is that an alternative to PR 66?
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
@magreenblatt Here is an alternative PR on GitHub which exposes CefOverridePath function: #4
Original comment by Czarek Tomczak (Bitbucket: Czarek, GitHub: Czarek).
@magreenblatt Have you took look at the new PR on GitHub? Please let me know if this solution is acceptable or whether I should create a new PR with this fix configurable with CefSettings.module_path option or similar.
@czarek See comments in the Github PR. I think CefOverridePath could be a good option with some safety modifications.
Original comment by Guillermo Zunino (Bitbucket: Guilllermo Zunino).
This issue is a critical for any integration of CEF with VM runtimes like java or python, to provide a seamless user experience that doesn't require manually copying files. I'm working on SWT java integration of CEF and facing this issue which force users to copy icudtl.dat and *.bin files to /usr/lib/jvm/java-8-oracle/jre/bin
Original comment by Joe Lauer (Bitbucket: jlauer, GitHub: jlauer).
I second @guilllermozunino and how this patch would be incredibly useful for integrations into Java. I am trying to get jcef packaged into a jar that gets expanded at runtime & loaded, but no matter what I do the icudtl.dat and *.bin files MUST reside where the java bin is. If I could provide my own path, I could then include everything in a single .jar, extract the contents at runtime, and then load up jcef.
Fixed in master revision 602c163 (bb).
Linux: Override DIR_ASSETS with libcef directory (see issue #1936)
→ <<cset 0b6ec3375115 (bb)>>
To explain the current behavior:
- PathProvider maps DIR_ASSETS to DIR_MODULE to FILE_MODULE.
- On Linux PathProviderPosix returns the exe path for FILE_MODULE.
- On Windows PathProviderWin returns the lib path for FILE_MODULE.
We should probably load resources (locales/* and *.pak files) from DIR_ASSETS instead of DIR_MODULE in GetResourcesFilePath on Linux.
The swiftshader binaries (swiftshader/*.so) are also currently loaded from DIR_MODULE in InitializeStaticEGLInternal.
The chrome-sandbox binary is currently loaded from DIR_EXE in SetuidSandboxHost::GetSandboxBinaryPath and configurable using the CHROME_DEVEL_SANDBOX environment variable. We can change this to DIR_ASSETS as well.
Original comment by Riku Palomäki (Bitbucket: riku_palomaki).
chrome-sandbox binary is not really used anymore, except maybe if you have some really ancient kernel. I wouldn't care about that.
chrome-sandbox binary is not really used anymore, except maybe if you have some really ancient kernel. I wouldn't care about that.
Thanks for pointing that out. The sandbox type is controlled in ZygoteHostImpl::Init. The chrome-sandbox binary will be used if the kernel doesn't support CLONE_NEWUSER or if it's not possible to immediately move to a new user namespace. CLONE_NEWUSER was added around kernel version 3.8 but I'm not sure about that second condition.
It looks like Ubuntu 14.04 includes 3.13 and Debian Jessie includes 3.16, so maybe we can stop shipping the chrome-sandbox binary?
I've filed this as issue #2630.
Linux: Load additional binaries from DIR_ASSETS (fixes issue #1936)
This adds .pak, locales/.pak , chrome-sandbox, libGLESv2.so, libEGL.so and
swiftshader/*.so to the list of binaries that will be loaded from the libcef.so
directory instead of the executable directory by default.
→ <<cset b3468451f545 (bb)>>
Linux: Load additional binaries from DIR_ASSETS (fixes issue #1936)
This adds .pak, locales/.pak , chrome-sandbox, libGLESv2.so, libEGL.so and
swiftshader/*.so to the list of binaries that will be loaded from the libcef.so
directory instead of the executable directory by default.
→ <<cset 2e39487ad6d4 (bb)>>
Linux: Load additional binaries from DIR_ASSETS (fixes issue #1936)
This adds .pak, locales/.pak , chrome-sandbox, libGLESv2.so, libEGL.so and
swiftshader/*.so to the list of binaries that will be loaded from the libcef.so
directory instead of the executable directory by default.
→ <<cset ee4b49f311d5 (bb)>>
Original comment by Henri Beauchamp (Bitbucket: Henri Beauchamp).
I just found out (with both CEF 74 and 75), that locales/*.pak still have to be located along the executable binay, and *not* along libcef.so…
I just found out (with both CEF 74 and 75), that locales/*.dat still have to be located along the executable directory, and *not* along libcef.so…
Do you mean locales/*.pak? Did you set CefSettings.locales_dir_path?
Original comment by Henri Beauchamp (Bitbucket: Henri Beauchamp).
Yes, *pak (I was editing my post when you replied to it… You are damned fast ! 😃 )…
And no, CefSettings.locales_dir_path is not set in Dullahan (the host executable using CEF, made by Second Life programmers). But even as the default path, should not it be the same path as for other .pak files (those not in the locales/ directory) ? After all, those locales/*.pak files are specific to libcef.so, not to the executable using it…
Linux: Load *.bin files from DIR_ASSETS (see issue #1936)
→ <<cset 9b9a9f359ec4 (bb)>>
From PR#265, without the above change loading will fail with the following errors when the application binary is not in the same directory as the .so:
[1031/074208.386065:WARNING:child_process_launcher_helper_posix.cc(119)] Ignoring invalid file v8_context_snapshot.bin
[1031/074208.386093:WARNING:child_process_launcher_helper_posix.cc(119)] Ignoring invalid file natives_blob.bin
Linux: Load *.bin files from DIR_ASSETS (see issue #1936)
→ <<cset 942899d2fc3d (bb)>>
- set component to "Framework-HasFix"
- changed state from "new" to "resolved"
- changed state from "resolved" to "open"
- changed state from "open" to "resolved"