Bazel build with LTO fails

Question

Bazel build with LTO fails

Closed this issue 5 months ago · 21 comments

When using the Bazel emscripten toolchain as outlined here, when building and linking with LTO via --copt="-flto" --linkopt="-flto", the linking step seems to fail because of the frozen cache setting in use here.

Reproducible so far on MacOS and Linux. This is an example error trace with a bit of internal nomenclature removed.

Linking of rule '<redacted>' failed (Exit 1): emcc_link.sh failed: error executing command external/emsdk/emscripten_toolchain/emcc_link.sh @bazel-out/wasm-opt-ST-53a11ea50094/bin/<path to target>-2.params
2021-04-29 13:02:20 
2021-04-29 13:02:20 Use --sandbox_debug to see verbose messages from the sandbox emcc_link.sh failed: error executing command external/emsdk/emscripten_toolchain/emcc_link.sh @bazel-out/wasm-opt-ST-53a11ea50094/bin/<path to target>-2.params
2021-04-29 13:02:20 
2021-04-29 13:02:20 Use --sandbox_debug to see verbose messages from the sandbox
2021-04-29 13:02:20 Traceback (most recent call last):
2021-04-29 13:02:20   File "emscripten_bin_linux/emscripten/emcc.py", line 3304, in <module>
2021-04-29 13:02:20     sys.exit(main(sys.argv))
2021-04-29 13:02:20   File "emscripten_bin_linux/emscripten/emcc.py", line 3297, in main
2021-04-29 13:02:20     ret = run(args)
2021-04-29 13:02:20   File "emscripten_bin_linux/emscripten/emcc.py", line 2105, in run
2021-04-29 13:02:20     extra_files_to_link += system_libs.calculate([f for _, f in sorted(temp_files)] + extra_files_to_link, link_as_cxx, forced=forced_stdlibs)
2021-04-29 13:02:20   File "emscripten_bin_linux/emscripten/tools/system_libs.py", line 1461, in calculate
2021-04-29 13:02:20     add_library(system_libs_map[forced])
2021-04-29 13:02:20   File "emscripten_bin_linux/emscripten/tools/system_libs.py", line 1452, in add_library
2021-04-29 13:02:20     libs_to_link.append((lib.get_path(), need_whole_archive))
2021-04-29 13:02:20   File "emscripten_bin_linux/emscripten/tools/system_libs.py", line 318, in get_path
2021-04-29 13:02:20     return shared.Cache.get_lib(self.get_filename(), self.build)
2021-04-29 13:02:20   File "emscripten_bin_linux/emscripten/tools/cache.py", line 130, in get_lib
2021-04-29 13:02:20     return self.get(name, *args, **kwargs)
2021-04-29 13:02:20   File "emscripten_bin_linux/emscripten/tools/cache.py", line 145, in get
2021-04-29 13:02:20     raise Exception('FROZEN_CACHE is set, but cache file is missing: %s' % shortname)
2021-04-29 13:02:20 Exception: FROZEN_CACHE is set, but cache file is missing: sysroot/lib/wasm32-emscripten/lto/libembind-rtti.a

Answer 1 · 2021-04-29T18:58:37.000Z

Sounds like have more stuff we might want to add the pre-built cache there.

Answer 2 · 2021-04-29T19:03:35.000Z

That makes sense. I believe the LTO builds use LTO-built variants of system libraries.

Any pointers on how to add things to the pre built cache?

Answer 3 · 2021-04-29T21:19:57.000Z

I think the bazel SDK is based on the emsdk binaries. emsdk itself ships just common configurations of libraries and does not set FROZEN_CACHE. So there are a few options:

Add more configs to emsdk builds
Create a separate archive that includes full set of libraries and add that to bazel SDK too
Remove FROZEN_CACHE from bazel sdk

@walkingeyerobot will probably have some idea about which makes sense here.

Answer 4 · 2021-05-03T16:04:37.000Z

Hi @walkingeyerobot , what do you think would be the best path forward here?

Answer 5 · 2021-05-03T17:31:17.000Z

Hey, sorry for the late reply!

I think we should go with option 1, adding more configs to the emsdk build. I don't think that will push the emsdk builds to unreasonable sizes, and I think it's intuitive for users to have both LTO and non-LTO builds work.

If emsdk maintainers are uncomfortable with this, we can certainly look into the other two options.

Answer 6 · 2021-05-03T17:31:31.000Z

cc @trybka

Answer 7 · 2021-05-04T16:35:37.000Z

That sounds good to me. What are the next steps here?

Answer 8 · 2021-05-18T23:03:57.000Z

Hi @walkingeyerobot @trybka @sbc100, sorry for the ping, is there any way I can help push this forward?

Answer 9 · 2021-05-18T23:29:10.000Z

I think the quickest solution would be to run embuilder build ALL --lto as part of the image build script.

You could probably get this working today (without landing any change here) by creating your own docker imagine based on the official one that just runs that extra command. This is very simple with docker because of that layering tech.

Answer 10 · 2021-05-19T19:06:51.000Z

Any way to do that with the bazel toolchain? I believe the bazel toolchain downloads its own release bundle to use, so running bazel within docker still wouldn't give you access to the libs built in the container? I guess we could ditch the toolchain for now and work off of the docker image?

Answer 11 · 2021-05-19T19:19:06.000Z

Ah sorry I forgot we were chatting about the bazel toolchain.

What happens if you simply remove FROZEN_CACHE from bazel/emscripten_toolchain/emscripten_config?

Answer 12 · 2021-05-31T19:50:17.000Z

This seems to work but requires having write permissions to ~/.emscripten_cache

Answer 13 · 2021-05-31T22:56:06.000Z

Do you not have write access to your own home directory? Or is an artifact of the way blaze tries to setup its hermetic environment?

Answer 14 · 2021-05-31T23:01:09.000Z

I think it's related to the hermetic environment, yeah. I think, at least in sandbox mode, that it's run this way intentionally. Haven't 100% confirmed though

Answer 15 · 2021-06-01T00:52:20.000Z

Confirmed. It works with --spawn_strategy=local but not sandboxed

Answer 16 · 2021-07-21T17:22:14.000Z

What are the next steps to move this issue forward?

Answer 17 · 2021-08-02T20:11:10.000Z

@sbc100 what builds the archives hosted at https://storage.googleapis.com/webassembly/emscripten-release-builds? can that be updated to include lto builds for future emscripten versions?

Answer 18 · 2021-08-02T22:04:14.000Z

The code that builds the releases lives at https://chromium.googlesource.com/emscripten-releases/ and runs on chromium CI infra at: https://ci.chromium.org/p/emscripten-releases

The process is still in flux but currently @dschuff (or one of us) has to manually trigger an LTO build for each release we bless.

Answer 19 · 2021-08-02T23:01:51.000Z

Wait, are we talking about building wasm LTO builds with a bazel toolchain, or using LTO-optimized compiler binaries?

Answer 20 · 2022-06-17T23:51:55.000Z

Wait, are we talking about building wasm LTO builds with a bazel toolchain, or using LTO-optimized compiler binaries?

The first option, so that bazel can compile with lto

Answer 21 · 2022-06-18T00:11:44.000Z

I think the quickest solution would be to run embuilder build ALL --lto as part of the image build script.

You could probably get this working today (without landing any change here) by creating your own docker imagine based on the official one that just runs that extra command. This is very simple with docker because of that layering tech.

I'm trying to work around this for my Org while we wait for upstream support (ideally as discussed in #971), I've done as you suggested:

FROM emscripten/emsdk:3.1.13

RUN ["/emsdk/upstream/emscripten/embuilder", "build", "ALL", "--lto"]

How do I turn this docker image into an archive compatible with the bazel toolchain?