proxy-wasm/proxy-wasm-cpp-host

[CI] macOS builds: "AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?"

lum1n0us opened this issue · 4 comments

lum1n0us commented

There is a common CI issue on macOS/x86-64 about pkgutil. ImpImporter triggered by 'rules_python"

ERROR: An error occurred during the fetch of repository 'fuzzing_py_deps':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_runner/f7b5b126cb65bf12475e[29](https://github.com/proxy-wasm/proxy-wasm-cpp-host/actions/runs/8042165716/job/21962477785#step:9:30)2acf07553d/external/rules_python/python/pip_install/pip_repository.bzl", line 171, column 13, in _pip_repository_impl
		fail("rules_python failed: %s (%s)" % (result.stdout, result.stderr))
Error in fail: rules_python failed:  (Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/private/var/tmp/_bazel_runner/f7b5b126cb65bf12[47](https://github.com/proxy-wasm/proxy-wasm-cpp-host/actions/runs/8042165716/job/21962477785#step:9:48)5e292acf07553d/external/rules_python/python/pip_install/extract_wheels/__init__.py", line 15, in <module>
    from python.pip_install.extract_wheels.lib import arguments, bazel, requirements
  File "/private/var/tmp/_bazel_runner/f7b5b126cb65bf12475e292acf07553d/external/rules_python/python/pip_install/extract_wheels/lib/bazel.py", line 9, in <module>
    from python.pip_install.extract_wheels.lib import namespace_pkgs, purelib, wheel
  File "/private/var/tmp/_bazel_runner/f7b5b126cb65bf12475e292acf07553d/external/rules_python/python/pip_install/extract_wheels/lib/purelib.py", line 6, in <module>
    from python.pip_install.extract_wheels.lib import wheel
  File "/private/var/tmp/_bazel_runner/f7b5b126cb65bf12475e292acf07553d/external/rules_python/python/pip_install/extract_wheels/lib/wheel.py", line 9, in <module>
    import pkg_resources
  File "/private/var/tmp/_bazel_runner/f7b5b126cb65bf12475e292acf07553d/external/pypi__setuptools/pkg_resources/__init__.py", line 2158, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

It turns out that pkgutil.ImpImporter, as a long-deprecated class, has been removed from python 3.12. A proper way of fixing is upgrade rules_python.

But if using the latest rules_python 0.31, another error will be meet.

ERROR: /root/.cache/bazel/_bazel_root/adb5dd13720c6f98da219c2f1ac0ff1a/external/rules_python/python/private/common/py_executable.bzl:840:12: name 'RunEnvironmentInfo' is not defined

It is because RunEnvironmentInfo is not exposed in Bazel < 5.3. Currently, proxy-wasm-cpp-host is using Build label: 5.2.0.

So I am kind of stuck here, shall I ?

  • try with another version of rules_python
  • just upgrade bazel environment
  • something else
PiotrSikora commented

Updating Bazel to 7.x and rules_python seems like a way to go. Thanks!

lum1n0us commented

Sorry for delayed response.

The new rules_python provides a different way for pip_install() and it beats me. Hope #389 can resolve it.

martijneken commented

Early finding: I can upgrade to Bazel 6.5.0 but with Bazel 7 (turning off bzlmod) I get wasm build errors. I may proceed with 6.5.0 for now (which matches Envoy).

Build errors seem to generally relate to toolchains + transitions:

  • --cxxopt gets lost in C++ wasm transition, leading to no type named 'string_view' in namespace 'std'
    • workaround is --action_env=BAZEL_CXXOPTS="-std=c++17"
  • Emscripten linker gets lost somewhere (see below) -- this transition is in proxy-wasm-cpp-sdk
  • Rust wasm complains (see further below) -- may be fixed with rules_rust update?
$ bazelisk build --verbose_failures --define engine=null --config=clang test/test_data:http_logging.wasm
INFO: Analyzed target //test/test_data:http_logging.wasm (0 packages loaded, 0 targets configured).
ERROR: /usr/local/google/home/mstevenson/git/proxy-wasm-cpp-host/test/test_data/BUILD:88:21: Linking test/test_data/proxy_wasm_http_logging failed: (Exit 1): clang failed: error executing CppLink command (from target //test/test_data:proxy_wasm_http_logging) 
  (cd /usr/local/google/home/mstevenson/.cache/bazel/_bazel_mstevenson/35e00757f2b858287fbaf6f5989ae0e8/sandbox/linux-sandbox/87/execroot/proxy_wasm_cpp_host && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-std=c++17' \
    CC=clang \
    CXX=clang++ \
    PWD=/proc/self/cwd \
    ZERO_AR_DATE=1 \
  /usr/lib/llvm-16/bin/clang @bazel-out/wasm-fastbuild-ST-27e13bb277c1/bin/test/test_data/proxy_wasm_http_logging-2.params)
# Configuration: 3f4fa744449c87a213b9b9765df0a7d5283d4371acdefe4a9785b7bd181e7ab7
# Execution platform: @@internal_platforms_do_not_use//host:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
clang: error: unsupported option '--no-entry'
clang: error: unsupported option '--js-library=external/proxy_wasm_cpp_sdk/proxy_wasm_intrinsics.js'
clang: error: unknown argument: '-sSTANDALONE_WASM'
clang: error: unknown argument: '-sEXPORTED_FUNCTIONS=_malloc'

$ bazelisk build --verbose_failures --define engine=null --config=clang test/test_data:random.wasm
ERROR: /usr/local/google/home/mstevenson/git/proxy-wasm-cpp-host/test/test_data/BUILD:77:17: in (an implicit dependency) attribute of wasi_rust_binary_rule rule //test/test_data:random.wasm: in $whitelist_function_transition attribute of wasi_rust_binary_rule rule //test/test_data:random.wasm: package group '@@bazel_tools//tools/whitelists/function_transition_whitelist:function_transition_whitelist' is misplaced here (they are only allowed in the visibility attribute). Since this rule was created by the macro 'wasm_rust_binary', the error might have been caused by the macro implementation
ERROR: /usr/local/google/home/mstevenson/git/proxy-wasm-cpp-host/test/test_data/BUILD:77:17: Analysis of target '//test/test_data:random.wasm' failed
ERROR: Analysis of target '//test/test_data:random.wasm' failed; build aborted
PiotrSikora commented

Drive-by comment, but copts = ["-std=c++17"] should be added to all Bazel C++ targets in C++ Host, since Bazel workspaces that depend on it won't use our .bazelrc, and instead need to force this flag themselves.

This works in Envoy right now, because it's also forcing C++17, but this should be fixed sooner, rather than later.