`//eventuals/grpc` breaks `cc_binary` targets built with `linkshared = True`

Question

`//eventuals/grpc` breaks `cc_binary` targets built with `linkshared = True`

Closed this issue 2 years ago · 2 comments

Given a minimal sample-binary.cc that defines an empty main function:

int main() {
  return 0;
}

These two binaries build:

cc_binary(
    name = "sample-eventuals-grpc-binary-default-link",
    srcs = ["sample-binary.cc"],
    copts = copts(),
    # Use default linking.
    deps = ["//eventuals/grpc"],
)

cc_binary(
    name = "sample-eventuals-grpc-binary-static-link",
    srcs = ["sample-binary.cc"],
    copts = copts(),
    # Use static linking (build libraries as `.a` files).
    linkstatic = True,
    deps = ["//eventuals/grpc"],
)

But this binary does not:

cc_binary(
    name = "sample-eventuals-grpc-binary-shared-link",
    srcs = ["sample-binary.cc"],
    copts = copts(),
    # Use dynamic linking (build libraries as `.so` files).
    linkshared = True,
    deps = ["//eventuals/grpc"],
)

The error is reproducible without //eventuals/grpc by depending on both @com_github_grpc_grpc//:grpc++ @com_github_jemalloc_jemalloc//:jemalloc together:

cc_binary(
    name = "sample-grpc-and-jemalloc-binary-shared-link",
    srcs = ["sample-binary.cc"],
    copts = copts(),
    linkshared = True,
    deps = [
        "@com_github_grpc_grpc//:grpc++",
        "@com_github_jemalloc_jemalloc//:jemalloc",
    ],
)

The error is about invalid relocations given the compiler's flags:

 /workspaces/respect/submodules/eventuals/test/BUILD.bazel:118:10: Linking test/libsample-grpc-and-eventuals-binary-shared-link.so failed: (Exit 1): clang failed: error executing command 
  (cd /home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/63/execroot/com_github_3rdparty_eventuals && \
  exec env - \
    PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
  /usr/lib/llvm-14/bin/clang @bazel-out/k8-fastbuild/bin/test/libsample-grpc-and-eventuals-binary-shared-link.so-2.params)
# Configuration: b74a02822c949d313fa6fb43644395877f96df5af4a497bac0102267e6c72476
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol 'malloc_conf'; recompile with -fPIC
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(jemalloc.o)
>>> referenced by jemalloc.c:955 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/jemalloc.c:955)
>>>               jemalloc.o:(malloc_conf_init_helper) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol 'malloc_message'; recompile with -fPIC
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(malloc_io.o)
>>> referenced by malloc_io.c:84 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/malloc_io.c:84)
>>>               malloc_io.o:(je_malloc_write) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol 'malloc_message'; recompile with -fPIC
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(malloc_io.o)
>>> referenced by malloc_io.c:627 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/malloc_io.c:627)
>>>               malloc_io.o:(je_malloc_vcprintf) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol 'malloc_message'; recompile with -fPIC
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(malloc_io.o)
>>> referenced by malloc_io.c:627 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/malloc_io.c:627)
>>>               malloc_io.o:(je_malloc_cprintf) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol 'malloc_message'; recompile with -fPIC
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(malloc_io.o)
>>> referenced by malloc_io.c:627 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/malloc_io.c:627)
>>>               malloc_io.o:(je_malloc_printf) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd_tls.h:51 (include/jemalloc/internal/tsd_tls.h:51)
>>>               tsd.o:(je_tsd_fetch_slow) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd_tls.h:52 (include/jemalloc/internal/tsd_tls.h:52)
>>>               tsd.o:(je_tsd_fetch_slow) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd_tls.h:54 (include/jemalloc/internal/tsd_tls.h:54)
>>>               tsd.o:(je_tsd_fetch_slow) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.c:0 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/tsd.c:0)
>>>               tsd.o:(je_tsd_fetch_slow) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.c:0 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/tsd.c:0)
>>>               tsd.o:(je_tsd_fetch_slow) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.c:0 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/tsd.c:0)
>>>               tsd.o:(je_tsd_fetch_slow) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.c:0 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/tsd.c:0)
>>>               tsd.o:(je_tsd_fetch_slow) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd_tls.h:51 (include/jemalloc/internal/tsd_tls.h:51)
>>>               tsd.o:(je_tsd_cleanup) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd_tls.h:52 (include/jemalloc/internal/tsd_tls.h:52)
>>>               tsd.o:(je_tsd_cleanup) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd_tls.h:54 (include/jemalloc/internal/tsd_tls.h:54)
>>>               tsd.o:(je_tsd_cleanup) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.h:353 (include/jemalloc/internal/tsd.h:353)
>>>               tsd.o:(je_malloc_tsd_boot0) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.c:430 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/tsd.c:430)
>>>               tsd.o:(je_malloc_tsd_boot0) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.c:430 (/home/vscode/.cache/dazel/eventuals/sandbox/processwrapper-sandbox/11/execroot/com_github_3rdparty_eventuals/bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_.build_tmpdir/src/tsd.c:430)
>>>               tsd.o:(je_malloc_tsd_boot0) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.h:354 (include/jemalloc/internal/tsd.h:354)
>>>               tsd.o:(je_malloc_tsd_boot0) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: relocation R_X86_64_TPOFF32 against je_tsd_tls cannot be used with -shared
>>> defined in bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a(tsd.o)
>>> referenced by tsd.h:353 (include/jemalloc/internal/tsd.h:353)
>>>               tsd.o:(je_malloc_tsd_boot1) in archive bazel-out/k8-fastbuild-ST-dd44be474a39/bin/external/com_github_jemalloc_jemalloc/jemalloc_preinstalled_make_/lib/libjemalloc.a

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //test:sample-grpc-and-eventuals-binary-shared-link failed to build
INFO: Elapsed time: 0.837s, Critical Path: 0.52s
INFO: 5 processes: 4 internal, 1 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

I discovered this when trying to use pybind11 (via https://github.com/pybind/pybind11_bazel#bazel-extensions-for-pybind11 ) to call C++ code from Python. This builds the wrapped C++ libraries as shared libraries, as can be seen in this example: https://github.com/cdleary/bazel_pybind_sample/blob/master/BUILD#L10

Sample PR that configures these binaries: #469

The break can be reproduced by running these commands on the xander.shared-wont-build branch:

dazel build //test:sample-eventuals-grpc-binary-shared-link

or:

dazel build //test:sample-grpc-and-jemalloc-binary-shared-link

Modifying our code in https://github.com/3rdparty/bazel-rules-jemalloc/blob/main/BUILD.bazel that defines @com_github_jemalloc_jemalloc//:jemalloc seems to affect this error: more on that in another comment!

Answer 1 · 2022-07-18T16:45:59.000Z

I can get the linkshared = True build to succeed if I comment out this out_static_libs property: https://github.com/3rdparty/bazel-rules-jemalloc/blob/c82c0c3856f07d53c1b76e89beeb8abab8c2d0ad/BUILD.bazel#L29-L34

And replace it with an out_shared_libs property:

    out_shared_libs = select({
        "@bazel_tools//src/conditions:windows": [
            "libjemalloc_so.lib",
        ],
        "//conditions:default": ["libjemalloc.so"],
    }),
    # out_static_libs = select({
    #     "@bazel_tools//src/conditions:windows": [
    #         "libjemalloc_a.lib",
    #     ],
    #     "//conditions:default": ["libjemalloc.a"],
    # }),

Simply adding the out_shared_libs property without commenting out the out_static_libs does not resolve the issue or change the produced error.

This change appears to make all 3 binaries build, and it passes eventuals tests. I don't know if it breaks anything because I don't know how to check whether jemalloc is correctly used.

Overall, this build issue makes me think that:
a) We should document why we're using any non-standard malloc replacement.
b) We should consider using tcmalloc instead of jemalloc, since tcmalloc supports being built by bazel out of the box: https://github.com/google/tcmalloc/blob/master/docs/quickstart.md#creating-your-build-file so we don't have to hand-roll bazel support (which the jemalloc team didn't want to add because it would be too hard to do correctly if cockroachdb/cockroach#57976 (comment) is correct)
c) We should consider replacing malloc at a binary level, not at a library level, so we're not trying to conflict with other malloc replacements (which gRPC might be doing)

I'm going to send out a PR to do something like switch to out_shared_libs.

Answer 2 · 2022-07-18T17:10:42.000Z

I expected that adding out_shared_libs would've helped, huh...
Looks like I picked the wrong week to quit drinking.