Build failing in master with --enable-static under MSys with the Visual Studio compiler
Closed this issue · 8 comments
The static build is failing in master with the cl compiler when the ifort compiler is also configured. This is because of a tangle of things and I don't know how the solution yet. Here are the commands to replicate. First,
C:\Users\tkral>"C:\Program Files (x86)\Intel\Composer XE 2015\bin\ifortvars.bat" intel64
C:\Users\tkral>set PATH=C:\msys64\usr\bin;%PATH%
C:\Users\tkral>bash
~> coinbrew fetch Clp
~> coinbrew build Clp --no-prompt --enable-msvc --enable-static --disable-shared
The eventual result of this is
make[1]: Entering directory '/home/tkral/Projects/build-static-msvc2017/Clp/master/src'
/bin/sh ../libtool --tag=CXX --mode=link /home/tkral/Projects/Clp/compile cl -nologo -EHs -DNDEBUG -O2 -MD -no-undefined -o libClp.la -rpath /home/tkral/Projects/dist-static/lib ClpCholeskyBase.lo ClpCholeskyDense.lo ClpConstraint.lo
ClpConstraintLinear.lo ClpConstraintQuadratic.lo Clp_C_Interface.lo ClpDualRowDantzig.lo ClpDualRowPivot.lo ClpDualRowSteepest.lo ClpDummyMatrix.lo ClpDynamicExampleMatrix.lo ClpDynamicMatrix.lo ClpEventHandler.lo ClpFactorization.lo ClpGubDynamicMatrix.lo ClpGubMatrix.lo ClpHelperFunctions.lo ClpInterior.lo ClpLinearObjective.lo ClpMatrixBase.lo ClpMessage.lo ClpModel.lo ClpNetworkBasis.lo ClpNetworkMatrix.lo ClpNonLinearCost.lo ClpNode.lo ClpObjective.lo ClpPackedMatrix.lo
ClpPlusMinusOneMatrix.lo ClpPredictorCorrector.lo ClpPdco.lo ClpPdcoBase.lo ClpLsqr.lo ClpPresolve.lo ClpPrimalColumnDantzig.lo ClpPrimalColumnPivot.lo ClpPrimalColumnSteepest.lo ClpQuadraticObjective.lo ClpSimplex.lo ClpSimplexDual.lo ClpSimplexNonlinear.lo ClpSimplexOther.lo ClpSimplexPrimal.lo ClpSolve.lo Idiot.lo IdiSolve.lo ClpCholeskyPardiso.lo ClpPESimplex.lo ClpPEPrimalColumnDantzig.lo ClpPEPrimalColumnSteepest.lo ClpPEDualRowDantzig.lo ClpPEDualRowSteepest.lo ClpCholeskyMumps.lo -L/home/tkral/Projects/dist-static/lib -lcoinmumps mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib -lCoinUtils mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib -lcoinasl -lcoinglpk
*** Warning: Linking the shared library libClp.la against the
*** static library mkl_intel_lp64.lib is not portable!
*** Warning: Linking the shared library libClp.la against the
*** static library mkl_sequential.lib is not portable!
*** Warning: Linking the shared library libClp.la against the
*** static library mkl_core.lib is not portable!
copying selected object files to avoid basename conflicts...
libtool: link: ln mkl_intel_lp64.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib || cp mkl_intel_lp64.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib
ln: failed to access 'mkl_intel_lp64.lib': No such file or directory
cp: cannot stat 'mkl_intel_lp64.lib': No such file or directory
libtool: link: ln mkl_sequential.lib .libs/libClp.lax/lt2-mkl_sequential.lib || cp mkl_sequential.lib .libs/libClp.lax/lt2-mkl_sequential.lib
ln: failed to access 'mkl_sequential.lib': No such file or directory
cp: cannot stat 'mkl_sequential.lib': No such file or directory
libtool: link: ln mkl_core.lib .libs/libClp.lax/lt3-mkl_core.lib || cp mkl_core.lib .libs/libClp.lax/lt3-mkl_core.lib
ln: failed to access 'mkl_core.lib': No such file or directory
cp: cannot stat 'mkl_core.lib': No such file or directory
libtool: link: lib -nologo -out:.libs/Clp.lib mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib .libs/libClp.lax/lt2-mkl_sequential.lib .libs/libClp.lax/lt3-mkl_core.lib ClpCholeskyBase.obj ClpCholeskyDense.obj ClpConstraint.obj ClpConstraintLinear.obj ClpConstraintQuadratic.obj Clp_C_Interface.obj ClpDualRowDantzig.obj ClpDualRowPivot.obj ClpDualRowSteepest.obj ClpDummyMatrix.obj ClpDynamicExampleMatrix.obj ClpDynamicMatrix.obj ClpEventHandler.obj ClpFactorization.obj ClpGubDynamicMatrix.obj ClpGubMatrix.obj ClpHelperFunctions.obj ClpInterior.obj ClpLinearObjective.obj ClpMatrixBase.obj ClpMessage.obj ClpModel.obj ClpNetworkBasis.obj ClpNetworkMatrix.obj ClpNonLinearCost.obj ClpNode.obj ClpObjective.obj ClpPackedMatrix.obj ClpPlusMinusOneMatrix.obj ClpPredictorCorrector.obj ClpPdco.obj ClpPdcoBase.obj ClpLsqr.obj ClpPresolve.obj ClpPrimalColumnDantzig.obj ClpPrimalColumnPivot.obj ClpPrimalColumnSteepest.obj ClpQuadraticObjective.obj ClpSimplex.obj ClpSimplexDual.obj ClpSimplexNonlinear.obj ClpSimplexOther.obj ClpSimplexPrimal.obj ClpSolve.obj Idiot.obj IdiSolve.obj ClpCholeskyPardiso.obj ClpPESimplex.obj ClpPEPrimalColumnDantzig.obj ClpPEPrimalColumnSteepest.obj ClpPEDualRowDantzig.obj ClpPEDualRowSteepest.obj ClpCholeskyMumps.obj
LINK : fatal error LNK1181: cannot open input file 'mkl_intel_lp64.lib'
make[1]: *** [Makefile:900: libClp.la] Error 157
make[1]: Leaving directory '/home/tkral/Projects/build-static-msvc2017/Clp/master/src'
make: *** [Makefile:757: all] Error 2
So it is trying to include the mkl libraries into the Clp library and cannot find them. Tracing the cause, it is because CoinUtils finds the mkl libraries during its configuration and adds them to the .pc file as follows.
Libs.private: mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib
This happens only because ifort has also been set up and so the path to the mkl libraries is added to the environment variable LIB. Hence, the libraries are included in the link line for pkg-config --libs coinutils and we end up with
CLPLIB_LFLAGS='-L/home/tkral/Projects/dist-static/lib -lcoinmumps mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib -lCoinUtils mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib -lcoinasl -lcoinglpk '
Meanwhile, in the src/Makefile.am for Clp (here), we have
libClp_la_LIBADD = $(CLPLIB_LFLAGS)
It appears that the LIB variable is not checked to find external libraries when linking a library, only when linking an executable, but this is not actually what's happening. The linker is finding the library, but libtool is itself trying to copy the library and this is the part that is failing (at least as far as I can tell right now).
This behavior does not arise with Osi, although it has the same setup and OSILIB_LFLAGS also includes the mkl libraries, because it seems Osi doesn't actually use any of the symbols in the library, so it is somehow ignored (or something like that, it's not very clear to me). During the Osi build, we get
*** Warning: Linking the shared library libOsiGlpk.la against the
*** static library mkl_intel_lp64.lib is not portable!
*** Warning: Linking the shared library libOsiGlpk.la against the
*** static library mkl_sequential.lib is not portable!
*** Warning: Linking the shared library libOsiGlpk.la against the
*** static library mkl_core.lib is not portable!
libtool: link: lib -nologo -out:.libs/OsiGlpk.lib mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib OsiGlpkSolverInterface.obj
mkl_core.lib(psdftsrecm7as_mc3.obj) : warning LNK4221: This object file does not define any previously undefined public
symbols, so it will not be used by any link operation that consumes this library
mkl_core.lib(psdftsrecm7as_mc.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
libtool: link: : .libs/OsiGlpk.lib
libtool: link: ( cd ".libs" && rm -f "libOsiGlpk.la" && cp -pR "../libOsiGlpk.la" "libOsiGlpk.la" )
and the library links just fine. It seems to be finding the library, but not trying to copy it.
There is another aspect to this, which is that the error did not happen when linking executables for the unit tests of CoinUtils and Osi. It's still not exactly clear to me what's going on there, but it is at least in part because we are using the .la files for linking those, e.g.,
unitTest_LDADD = ../src/libCoinUtils.la
and in libCoinUtils.la, we have
# Libraries that this one depends upon.
dependency_libs=' -L/home/tkral/Projects/dist-static/lib /home/tkral/Projects/dist-static/lib/libcoinasl.la /home/tkral/Projects/dist-static/lib/libcoinglpk.la'
so the mkl libraries are not linked. I suppose they are not needed for the CoinUtils unit test, but the autotools doesn't know this. Perhaps it is assuming that the mkl libraries are being rolled into the CoinUtils library, so those are no longer explicit dependencies. Anyway, this opens up a bunch of questions.
- Is libtool really trying to include a copy of the mkl libraries into the libraries that depend on them or is it just trying to make a copy for some reason?
- Depending on the answer to the above, should the mkl libraries really be copied into each of the COIN libraries that depends on them?
- Does this mean that all of the dependent libraries are being copied into other libraries by libtool (I hope not)?
- Shouldn't the .pc files and .la files be specifying the same dependencies?
- Should we not be using the
*LIB_LFLAGSvariables for figuring out what libraries to link when creating a library (at least with Visual Studio). We use.lafiles for linking executables, but I guess those cannot be used for linking libraries, since we use_LDADDin one case and_LIBADDin the other case.
Anyway, this seems to be a thread that I tugged that's unraveling a lot of stuff. I'm still sorting through it, but thought I would get some folks involved (@svigerske @LouHafer )who know more about how the new build system hangs together before going much further.
Yes, libtool copies the MKL into other libs when doing static builds. I think that also happened with BuildTools 0.8 sometimes.
libtool tries to explain why it's doing this with copying selected object files to avoid basename conflicts....
So there is some check in libtool, which I'm not really understanding at the moment, but that might explain why it doesn't always do that.
The way how the MKL libs are named and presented in the linker flags may prevent libtool from adding them to the dependency_libs in the .la file.
I'm not sure that there is a good way to link the unittest of CoinUtils with the COINUTILSLIB_LFLAGS. So you suggest to do
unitTest_LDADD = -L../src -lCoinUtils $(COINUTILSLIB_LFLAGS)
But the CoinUtils library may sit in ../src/.libs, so that libtool will either not find it, or still picks up ../src/libCoinUtils.la. And since $(COINUTILSLIB_LFLAGS) has been recorded in the dependency_libs of libCoinUtils.la (except for MKL, it seems), dependencies would be duplicated.
Maybe one could extend the Lapack check to actually figure out the path to the MKL libs and add an appropriate -L flag or specify the libraries with full paths, so that libtool will find them for the Clp build.
Well, with that change, linking libOsiClp now stops with an error from link.exe, saying it cannot link more than 65535 objects into one library :)
Adding full paths is a good thought, but it's not simply that the path to the libraries is completely unknown. They are only in play to begin with because Visual Studio does find the libraries from the path in the LIB variable and this is how they got detected in the first place. In config.log for CoinUtils, we have
configure:18536: checking for function dsyev_ in mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib
configure:18552: /home/tkral/Projects/CoinUtils/compile cl -o conftest.exe -nologo -EHs -DNDEBUG -O2 -MD conftest.cpp mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib >&5
conftest.cpp
configure:18552: $? = 0
configure:18560: result: yes
So it works to just throw those libraries on the command line when linking an executable using compile. It just doesn't work to do it with libtool. The fundamental issue appears to me to be that the Visual Studio lib command doesn't seem to be checking the LIB variable for the path to libraries. However, according to this:
https://docs.microsoft.com/en-us/cpp/build/reference/libpath-additional-libpath?view=vs-2019
it is supposed to use LIB to get its search path. It seems this shouldn't be an issue. Perhaps the LIB variable is being over-written somewhere. I guess I can just play with libtool a bit and try to get it to print out the value of the LIB variable. what is happening doesn't make sense to me.
Well, looking at the output above again, there are some errors even before libtool calls the lib command. I don't know what's going on... it seems to be a libtool issue, but we can't be the first to face it.
I think lib is also fine with looking into $LIB. But the command that was failing was
libtool: link: ln mkl_intel_lp64.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib || cp mkl_intel_lp64.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib
ln: failed to access 'mkl_intel_lp64.lib': No such file or directory
cp: cannot stat 'mkl_intel_lp64.lib': No such file or directory
And I think that this is run by libtool and neither libtool nor ln look into $LIB.
So just having mkl_intel_lp64.lib as a flag that is passed to the compiler or linker is fine, but when libtool says "hey, this is a static library, so I can unpack it", then things go wrong.
Yeah, I see that the linking failed there, but then it appears to continue on and call lib later on with those libraries listed on the command line. The fatal error is actually from lib itself.
libtool: link: lib -nologo -out:.libs/Clp.lib mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib .libs/libClp.lax/lt1-mkl_intel_lp64.lib .libs/libClp.lax/lt2-mkl_sequential.lib .libs/libClp.lax/lt3-mkl_core.lib ClpCholeskyBase.obj ClpCholeskyDense.obj ClpConstraint.obj ClpConstraintLinear.obj ClpConstraintQuadratic.obj Clp_C_Interface.obj ClpDualRowDantzig.obj ClpDualRowPivot.obj ClpDualRowSteepest.obj ClpDummyMatrix.obj ClpDynamicExampleMatrix.obj ClpDynamicMatrix.obj ClpEventHandler.obj ClpFactorization.obj ClpGubDynamicMatrix.obj ClpGubMatrix.obj ClpHelperFunctions.obj ClpInterior.obj ClpLinearObjective.obj ClpMatrixBase.obj ClpMessage.obj ClpModel.obj ClpNetworkBasis.obj ClpNetworkMatrix.obj ClpNonLinearCost.obj ClpNode.obj ClpObjective.obj ClpPackedMatrix.obj ClpPlusMinusOneMatrix.obj ClpPredictorCorrector.obj ClpPdco.obj ClpPdcoBase.obj ClpLsqr.obj ClpPresolve.obj ClpPrimalColumnDantzig.obj ClpPrimalColumnPivot.obj ClpPrimalColumnSteepest.obj ClpQuadraticObjective.obj ClpSimplex.obj ClpSimplexDual.obj ClpSimplexNonlinear.obj ClpSimplexOther.obj ClpSimplexPrimal.obj ClpSolve.obj Idiot.obj IdiSolve.obj ClpCholeskyPardiso.obj ClpPESimplex.obj ClpPEPrimalColumnDantzig.obj ClpPEPrimalColumnSteepest.obj ClpPEDualRowDantzig.obj ClpPEDualRowSteepest.obj ClpCholeskyMumps.obj
LINK : fatal error LNK1181: cannot open input file 'mkl_intel_lp64.lib'
fatal error LINK 1181 is an error from lib, not libtool. Regardless of the earlier error, this seems like it should work. I'll try later to just call lib directly on the command line and see if I can figure out what's going on.
Weird when I re-run make by hanad, I now get
LINK : fatal error LNK1181: cannot open input file '.libs/libClp.lax/lt1-mkl_intel_lp64.lib'
If I run the lib command itself directly (removing reference to .libs/libClp.lax/lt1-mkl*, I get
$ lib -nologo -out:.libs/Clp.lib mkl_intel_lp64.lib mkl_sequential.lib mkl_core.lib ClpCholeskyBase.obj ClpCholeskyDense.o
bj ClpConstraint.obj ClpConstraintLinear.obj ClpConstraintQuadratic.obj Clp_C_Interface.obj ClpDualRowDantzig.obj ClpDualR
owPivot.obj ClpDualRowSteepest.obj ClpDummyMatrix.obj ClpDynamicExampleMatrix.obj ClpDynamicMatrix.obj ClpEventHandler.obj
ClpFactorization.obj ClpGubDynamicMatrix.obj ClpGubMatrix.obj ClpHelperFunctions.obj ClpInterior.obj ClpLinearObjective.o
bj ClpMatrixBase.obj ClpMessage.obj ClpModel.obj ClpNetworkBasis.obj ClpNetworkMatrix.obj ClpNonLinearCost.obj ClpNode.obj
mkl_core.lib(psdftsrecm7as_mc3.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
mkl_core.lib(psdftsrecm7as_mc.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
but the library is created just fine! This is he same kind of output I saw in Osi, where things eventually went through fine, despite the warnings. So something is happening here that is a bit different than what is happening in Osi, but I don't see what yet.
I just updated my comment above, I had made a stupid mistake in what I posted earlier (running the lib command in the wrong directory.