Linking Error with lib/Dialect/Poly/libMLIRPoly.dylib in MLIR Project

Question

Linking Error with lib/Dialect/Poly/libMLIRPoly.dylib in MLIR Project

anniezfy opened this issue a year ago · 13 comments

Hi~, I tried to build this tutorial as below commit version and also build corresponding correct external mlir project
I am encountering linking errors when building a shared library (lib/Dialect/Poly/libMLIRPoly.dylib). The error occurs during the linking phase and appears to be related to undefined symbols in the MLIR framework.
For example:

Undefined symbols:
mlir::Diagnostic::operator<<(llvm::Twine&&), referenced from: ...
mlir::ValueRange::dereference_iterator(llvm::PointerUnion<mlir::Value const*, mlir::OpOperand*, mlir::detail::OpResultImpl*> const&, long), referenced from:

Answer 1 · 2023-11-19T06:35:03.000Z

[18/34] Linking CXX shared library lib/Transform/Arith/libMulToAdd.dylib
FAILED: lib/Transform/Arith/libMulToAdd.dylib
: && /Library/Developer/CommandLineTools/usr/bin/c++ -g -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk -mmacosx-version-min=13.5 -dynamiclib -Wl,-headerpad_max_install_names -Wl,-dead_strip -o lib/Transform/Arith/libMulToAdd.dylib -install_name @rpath/libMulToAdd.dylib lib/Transform/Arith/CMakeFiles/obj.MulToAdd.dir/MulToAdd.cpp.o -Wl,-rpath,@loader_path/../lib -Wl,-rpath,/Users/anniezfy/MLIR_tutorial_2/mlir-tutorial/externals/llvm-project/build/./lib /Users/anniezfy/MLIR_tutorial_2/mlir-tutorial/externals/llvm-project/build/lib/libLLVMSupport.a -lm /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk/usr/lib/libz.tbd /usr/local/lib/libzstd.dylib /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk/usr/lib/libcurses.tbd /Users/anniezfy/MLIR_tutorial_2/mlir-tutorial/externals/llvm-project/build/lib/libLLVMDemangle.a && :
ld: Undefined symbols:
mlir::OperationState::OperationState(mlir::Location, mlir::OperationName), referenced from:
mlir::arith::ConstantOp mlir::OpBuilder::create<mlir::arith::ConstantOp, mlir::IntegerAttr>(mlir::Location, mlir::IntegerAttr&&) in MulToAdd.cpp.o
mlir::arith::MulIOp mlir::OpBuilder::create<mlir::arith::MulIOp, mlir::Value&, mlir::arith::ConstantOp&>(mlir::Location, mlir::Value&, mlir::arith::ConstantOp&) in MulToAdd.cpp.o
mlir::arith::AddIOp mlir::OpBuilder::create<mlir::arith::AddIOp, mlir::arith::MulIOp&, mlir::arith::MulIOp&>(mlir::Location, mlir::arith::MulIOp&, mlir::arith::MulIOp&) in MulToAdd.cpp.o
mlir::arith::AddIOp mlir::OpBuilder::create<mlir::arith::AddIOp, mlir::arith::MulIOp&, mlir::Value&>(mlir::Location, mlir::arith::MulIOp&, mlir::Value&) in MulToAdd.cpp.o
mlir::OperationState::~OperationState(), referenced from:
mlir::arith::ConstantOp mlir::OpBuilder::create<mlir::arith::ConstantOp, mlir::IntegerAttr>(mlir::Location, mlir::IntegerAttr&&) in MulToAdd.cpp.o
mlir::arith::MulIOp mlir::OpBuilder::create<mlir::arith::MulIOp, mlir::Value&, mlir::arith::ConstantOp&>(mlir::Location, mlir::Value&, mlir::arith::ConstantOp&) in MulToAdd.cpp.o
mlir::arith::AddIOp mlir::OpBuilder::create<mlir::arith::AddIOp, mlir::arith::MulIOp&, mlir::arith::MulIOp&>(mlir::Location, mlir::arith::MulIOp&, mlir::arith::MulIOp&) in MulToAdd.cpp.o
mlir::arith::AddIOp mlir::OpBuilder::create<mlir::arith::AddIOp, mlir::arith::MulIOp&, mlir::Value&>(mlir::Location, mlir::arith::MulIOp&, mlir::Value&) in MulToAdd.cpp.o
mlir::PatternBenefit::PatternBenefit(unsigned int), referenced from:
mlir::tutorial::PowerOfTwoExpand::PowerOfTwoExpand(mlir::MLIRContext*) in MulToAdd.cpp.o
mlir::tutorial::PeelFromMul::PeelFromMul(mlir::MLIRContext*) in MulToAdd.cpp.o
mlir::RewritePattern::anchor(), referenced from:
vtable for mlir::tutorial::PowerOfTwoExpand in MulToAdd.cpp.o
vtable for mlir::OpRewritePatternmlir::arith::MulIOp in MulToAdd.cpp.o
vtable for mlir::detail::OpOrInterfaceRewritePatternBasemlir::arith::MulIOp in MulToAdd.cpp.o
vtable for mlir::tutorial::PeelFromMul in MulToAdd.cpp.o

Answer 2 · 2023-11-20T04:11:33.000Z

It looks like you're using the CMake build, which I'm not as familiar with. Are you able to build it with bazel? I just want to know so that I can tell if it's a system-wide problem (perhaps related to the OS) versus a problem with my build files.

Answer 3 · 2023-11-20T04:13:50.000Z

I think the cmake setup here also gives you the flexibility to choose a different commit than what is used in the tutorial. Can you try navigating to externals/llvm-project and running git checkout f778eafdd878e8b11ad76f9e0a312ce7791a7481 (which is the same commit that the CI tests with), then trying to rebuild MLIR and this project?

Answer 4 · 2023-11-21T02:00:16.000Z

Thanks for your suggestion. I rebuilt the entire project with bazel successfully, but failure still comes since using CMake build. Errors are generated during the linking phase

Answer 5 · 2023-11-21T02:09:27.000Z

I'm not a cmake expert but I will take a look and leave this issue open for now. I imagine the issue is related to the C compiler on Mac, since the CI runs Linux and works fine.

Answer 6 · 2023-11-21T04:29:06.000Z

👌, Thanks for your reply~ I‘d like to build on Linux

Answer 7 · 2023-11-21T06:02:46.000Z

I guess it is probably Mac OS compiler problems, the project also runs successfully on the Linux platform

Answer 8 · 2023-11-21T14:38:40.000Z

I am experiencing locally the same problem of missing symbols at linking time (macOS arm64). I looked a bit into the issue and noticed that if I remove the or-tools dependency linking goes well (here the diff of what i removed: https://gist.github.com/vguerra/954a8b77ae2f97b1304d3afc9139349f ).

If I simply add this to the root CMakeLists.txt I start to see linking issues:

message(STATUS "Fetching or-tools...")
include(FetchContent)
FetchContent_Declare(
  or-tools
  GIT_REPOSITORY https://github.com/google/or-tools.git
  GIT_TAG        main
)
FetchContent_MakeAvailable(or-tools)
message(STATUS "Done fetching or-tools")

Answer 9 · 2023-11-22T08:53:48.000Z

OK,thanks for your tips

Answer 10 · 2023-11-22T16:24:21.000Z

Hmmm I would have hoped FetchContent would be easier to use than a submodule, but it seems not to be.

Answer 11 · 2023-11-23T15:08:37.000Z

I kept digging a bit on this build issue and had cmake/ninja dump the commands being executed and noticed that libs would be linked differently depending on whether you include or-tools or not.

I checked the or-tools repo for similar issues and came to google/or-tools#3709 .. specifically google/or-tools#3709 (comment) .. so turns out that turning off shared libs helps.

Configuring the project as follows would allow you to compile successfuly:

...
cmake -G $BUILD_SYSTEM .. \
    -DLLVM_DIR="$LLVM_BUILD_DIR/lib/cmake/llvm" \
    -DMLIR_DIR="$LLVM_BUILD_DIR/lib/cmake/mlir" \
    -DBUILD_DEPS="ON" \
    -DBUILD_SHARED_LIBS=OFF \ 
    -DCMAKE_BUILD_TYPE=Debug 
...

@anniezfy would it be possible to verify that this works for you? 🙏 😊

Answer 12 · 2023-11-23T16:47:40.000Z

@vguerra I compile this project again with the new cmake instructions It finally works successfully!!

Answer 13 · 2023-11-24T13:02:09.000Z

hi @j2kun .. I tested this as well for the CI build here https://github.com/vguerra/mlir-tutorial/actions/runs/6980308205 and works for the build on linux, so maybe the cmake action can be updated as well as the instructions in the readme. wdyt?