Compilation failures since 0.3.3 due to missing header
mhlakhani opened this issue · 10 comments
We use this crate to fuzz libra (https://github.com/libra/libra) - and it's been working really well, thank you for this integration!
We tried upgrading to 0.3.3 a while back and had to revert, because the build was breaking on certain machines which don't have LLVM source on the include path. I am not sure how to fix this, but I wanted to file an issue in the hopes someone can hepl fix it.
As far as I can tell, this is due to an upstream issue. 0.3.3 includes an updated version of libfuzzer (0493bb8) which most specifically includes llvm/llvm-project@4548236
This includes a file that's not in the libfuzzer source tree, and I don't see this file being present in this source repo.
I'm not super familiar with how the build.rs scripts work under the hood, but it seems like we need to ensure that this file is present in the repo and appropriately passed to the include paths for the compiler.
If it's helpful, here are the logs from our system:
OPT_LEVEL = Some("3")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = Some("fxsr,mmx,sse,sse2")
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerCrossOver.o" "-c" "libfuzzer/FuzzerCrossOver.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerDataFlowTrace.o" "-c" "libfuzzer/FuzzerDataFlowTrace.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerDriver.o" "-c" "libfuzzer/FuzzerDriver.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerExtFunctionsDlsym.o" "-c" "libfuzzer/FuzzerExtFunctionsDlsym.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerExtFunctionsWeak.o" "-c" "libfuzzer/FuzzerExtFunctionsWeak.cpp"
exit code: 0
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerExtFunctionsWindows.o" "-c" "libfuzzer/FuzzerExtFunctionsWindows.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerExtraCounters.o" "-c" "libfuzzer/FuzzerExtraCounters.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerFork.o" "-c" "libfuzzer/FuzzerFork.cpp"
exit code: 0
exit code: 0
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerIO.o" "-c" "libfuzzer/FuzzerIO.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerIOPosix.o" "-c" "libfuzzer/FuzzerIOPosix.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerIOWindows.o" "-c" "libfuzzer/FuzzerIOWindows.cpp"
exit code: 0
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerInterceptors.o" "-c" "libfuzzer/FuzzerInterceptors.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerLoop.o" "-c" "libfuzzer/FuzzerLoop.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerMain.o" "-c" "libfuzzer/FuzzerMain.cpp"
exit code: 0
cargo:warning=libfuzzer/FuzzerInterceptors.cpp:30:45: fatal error: sanitizer/common_interface_defs.h: No such file or directory
cargo:warning= #include <sanitizer/common_interface_defs.h>
cargo:warning= ^
cargo:warning=compilation terminated.
exit code: 1
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerMerge.o" "-c" "libfuzzer/FuzzerMerge.cpp"
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
--- stderr
error occurred: Command "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerInterceptors.o" "-c" "libfuzzer/FuzzerInterceptors.cpp" with args "c++" did not execute successfully (status code exit code: 1).
Hmm, the most relevant thing is maybe https://github.com/rust-fuzz/libfuzzer/blob/master/update-libfuzzer.sh
As you can see we check out the llvm repo and just copy the libfuzzer dir.
I could have sworn the libfuzzer docs used to claim you could just build it with cc *.c but I don't see that in the docs any longer.
@alex my understanding of that script is that it will copy over the files from upstream, but doesn't copy over that additional header. Fixing that wouldn't be sufficient (I think) since we also need to update https://github.com/rust-fuzz/libfuzzer/blob/master/build.rs to somehow get this on the path.
I could have sworn the libfuzzer docs used to claim you could just build it with cc *.c but I don't see that in the docs any longer.
I also recall seeing that, but I can't find it anymore - and I think https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/fuzzer/build.sh also is proof of that assumption.
I am happy to report this on the libfuzzer mailing list instead if you think that's a better place for this fix.
This is getting fixed upstream: https://reviews.llvm.org/D86981
Once that's landed we can pull in the changes and do a release if we want
Looks like the llvm fix is merged. Can we release now?
i will look into doing a release now
rust-fuzz/libfuzzer release is out. going to bump in cargo-fuzz
oh right, cargo-fuzz itself doesn't depend on libfuzzer, the generated fuzzing cargo projects do. hm i guess everyone who uses cargo-fuzz will eventually need to cargo update? a little unfortunate
in any case, this should be resolved now! just bump libfuzzer wherever you're using it