Failed to build czmq on Windows via CMake

[myd7349]

Hey, guys! I am back again.

I am trying to add czmq to vcpkg these days(microsoft/vcpkg#4979). When building czmq with CMake, I encountered several problems.

1. CMAKE_CURRENT_SOURCE_DIR isn't in CMAKE_MODULE_PATH.

   https://github.com/zeromq/czmq/blob/master/CMakeLists.txt#L19

   so Findlibzmq.cmake, Findlibsodium.cmake, .etc. may not be called.

2. When built as a static library, libzmq will set an additional compiler flag -DZMQ_STATIC through set_target_properties:

   https://github.com/zeromq/libzmq/blob/master/CMakeLists.txt#L1060

   so if we want to build czmq as a static library on Windows, we should define ZMQ_STATIC manually(Maybe we can define it in Findlibzmq.cmake). Otherwise, the build will fail: zeromq/libzmq#3318

3. Findlibzmq.cmake is not be able to find libzmq on Windows.

   libzmq dll/lib built with MSVC is named using the Boost convention(something like libzmq-mt-4_3_1.lib, libzmq-mt-4_3_1.dll). So find_library (LIBZMQ_LIBRARIES NAMES zmq HINTS ${PC_LIBZMQ_LIBRARY_HINTS}) will not work.

   Here is a patch for solving this problem: https://bitbucket.org/ignitionrobotics/ign-transport/pull-requests/19/detec-zmq-under-windows-using-a-findzmq/diff

4. Findlibsodium.cmake will not found libsodium on Windows

   When built with MSVC, libsodium will generate a library named libsodium.dll/lib. In this case, find_library(LIBSODIUM_LIBRARY NAMES sodium) is not able to find the library. I found that find_library(LIBSODIUM_LIBRARY NAMES sodium libsodium) works.

   By the way, Findlibcurl.cmake has this problem too.

5. Maybe we can add some options like:

   WITH_LIBSODIUM
WITH_LIBCURL
WITH_LZ4
WITH_UUID
BUILD_TESTS (For example: googleapis/google-cloud-cpp#1617)
   BUILD_TOOLS (zmakecret)

[ras0219-msft]

We can (and should) fix (2) upstream in vcpkg by encoding ZMQ_STATIC into the zmq headers.

[Deleted user]

You don't need to add BUILD_TESTS option. include(CTest) automatically calls enable_testing() and creates a BUILD_TESTING option for you. Depending on how you've structured your CMakeLists, you may need to wrap certain blocks in IF (BUILD_TESTING).

[myd7349]

@ras0219-msft Just like this?

if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
    file(READ ${CURRENT_PACKAGES_DIR}/include/zmq.h _contents)
    string(REPLACE "defined ZMQ_STATIC" "1 //defined ZMQ_STATIC" _contents "${_contents}")
    file(WRITE ${CURRENT_PACKAGES_DIR}/include/zmq.h "${_contents}")
endif()

I am considering to add these line to zeromq/portfile.cmake through microsoft/vcpkg#4214.

@remyabel Thanks for telling me that. :-)

[bluca]

Note that libsodium is not a dependency of CZMQ, so there's no need for an option for it

[stale]

This issue has been automatically marked as stale because it has not had recent activity for 90 days. It will be closed if no further activity occurs within 21 days. Thank you for your contributions.

[myd7349]

Do not close yet. I'd like to tackle this issue.

[myd7349]

A quick fix of this problem:

First, build and install libzmq on Windows:

cd libzmq
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=C:/libzmq
cmake --build . --target install

Then, clone czmq and replace Findlibzmq.cmake with this one: https://github.com/myd7349/Ongoing-Study/blob/master/cpp/CMake/find_libzmq_ng/Findlibzmq.cmake

cd czmq
mkdir build
cd build
cmake .. -DCMAKE_PREFIX_PATH=C:/libzmq

The CI configuration file has this line: https://github.com/zeromq/czmq/blob/master/appveyor.yml#L53
That is, it renames libs like libzmq-v141-mt-gd-4_3_2.lib to zmq.lib.

[myd7349]

Findlibzmq.cmake is generated by these code: https://github.com/zeromq/zproject/blob/master/zproject_cmake.gsl#L640-L697 via gsl -target:* project.xml.

[myd7349]

I think we may have three ways to fix the libzmq not found issue on Windows:

1. Remove <use project = "libzmq" /> from czmq/project.xml, and maintain Findlibzmq.cmake seperately. Then we can fix this problem by patching Findlibzmq.cmake manually.
2. Change these code https://github.com/zeromq/zproject/blob/master/zproject_cmake.gsl#L640-L697 to handle zmq specially so that a special version of Findlibzmq.cmake is generated like the one I gave above.
3. Prefer ZeroMQ's CMake target instead of Findlibzmq.cmake.
   ZeroMQ itself provides a CMake target(libzmq for a shared build, and libzmq-static for a static build). So we may consider replacing Findlibzmq.cmake with find_package(ZeroMQ CONFIG REQUIRED).
   This solution also require us to remove <use project = "libzmq" /> from czmq/project.xml, and then add find_package(ZeroMQ CONFIG REQUIRED) to this file: https://github.com/zeromq/zproject/blob/master/zproject_cmake.gsl

I personally think solution 3 is the best, since it uses modern cmake.

Update:
I noticed the existance of this #1939. So maybe I will fix this problem through solution 1 or 2.

[eplumer]

To close out on my question for #2004 which was redirected here I have attached the manual steps that I got working on my VS 2013 system. Thanks for pointing me to he appveyor.yml file for hints:

set ZMQ_ROOT=path_to_my_project_root
set CMAKE_GENERATOR="Visual Studio 12 2013 Win64"
set LIBZMQ_SOURCEDIR=%ZMQ_ROOT%\libzmq
set LIBZMQ_BUILDDIR=%LIBZMQ_SOURCEDIR%\BUILD
set CZMQ_SOURCEDIR=%ZMQ_ROOT%\czmq
set CZMQ_BUILDDIR=%CZMQ_SOURCEDIR%\BUILD

rmdir /s /q %LIBZMQ_BUILDDIR%
mkdir %LIBZMQ_BUILDDIR%
cd %LIBZMQ_BUILDDIR%

cmake -D CMAKE_CXX_FLAGS_RELEASE="/MT" -D CMAKE_CXX_FLAGS_DEBUG="/MTd" -G %CMAKE_GENERATOR% %LIBZMQ_SOURCEDIR%

C:\Progra~2\MSBuild\12.0\Bin\msbuild /v:minimal /maxcpucount:%NUMBER_OF_PROCESSORS% /p:Configuration=Release libzmq.vcxproj

C:\Progra~2\MSBuild\12.0\Bin\msbuild /v:minimal /maxcpucount:%NUMBER_OF_PROCESSORS% /p:Configuration=Debug libzmq.vcxproj

copy /Y %LIBZMQ_BUILDDIR%\lib\Release\libzmq-v120-mt-4_3_1.lib %LIBZMQ_BUILDDIR%\lib\Release\zmq.lib
copy /Y %LIBZMQ_BUILDDIR%\lib\Debug\libzmq-v120-mt-gd-4_3_1.lib %LIBZMQ_BUILDDIR%\lib\Debug\zmq.lib

rmdir /s /q %CZMQ_BUILDDIR%
mkdir %CZMQ_BUILDDIR%
cd %CZMQ_BUILDDIR%

cmake -G %CMAKE_GENERATOR% -D CMAKE_INCLUDE_PATH="%LIBZMQ_SOURCEDIR%\include" -D CMAKE_LIBRARY_PATH="%LIBZMQ_BUILDDIR%\lib\Release" -D CMAKE_C_FLAGS_RELEASE="/MT" -D CMAKE_CXX_FLAGS_RELEASE="/MT" -D CMAKE_C_FLAGS_DEBUG="/MTd" %CZMQ_SOURCEDIR%

Open Visual Studio 2013 and Rebuild "Release | x64" for "%CZMQ_BUILDDIR%\czmq.sln"

copy /Y %LIBZMQ_BUILDDIR%\bin\Release\libzmq-v120-mt-4_3_1.dll %CZMQ_BUILDDIR%\Release\

cmake -G %CMAKE_GENERATOR% -D CMAKE_INCLUDE_PATH="%LIBZMQ_SOURCEDIR%\include" -D CMAKE_LIBRARY_PATH="%LIBZMQ_BUILDDIR%\lib\Debug" -D CMAKE_C_FLAGS_RELEASE="/MT" -D CMAKE_CXX_FLAGS_RELEASE="/MT" -D CMAKE_C_FLAGS_DEBUG="/MTd" %CZMQ_SOURCEDIR%

Open Visual Studio 2013 and Rebuild "Debug | x64" for "%CZMQ_BUILDDIR%\czmq.sln"

copy /Y %LIBZMQ_BUILDDIR%\bin\Debug\libzmq-v120-mt-gd-4_3_1.dll %CZMQ_BUILDDIR%\Debug\
copy /Y %LIBZMQ_BUILDDIR%\bin\Debug\libzmq-v120-mt-gd-4_3_1.pdb %CZMQ_BUILDDIR%\Debug\

[myd7349]

Finally, I am able to build czmq (and almost all of its dependencies) successfully in vcpkg: microsoft/vcpkg#4979
czmq depends on many 3rd party libraries (optional or required), and it is hard to handle all these dependencies correctly with find_library.

ZeroMQ, for example, can use libsodium optionally. And ZeroMQ's CMake module exports a target named libzmq (or libzmq-static with a static build). The exported CMake target file may looks like this (Windows, Debug mode):

#----------------------------------------------------------------
# Generated CMake target import file for configuration "Debug".
#----------------------------------------------------------------

# Commands may need to know the format version.
set(CMAKE_IMPORT_FILE_VERSION 1)

# Import target "libzmq" for configuration "Debug"
set_property(TARGET libzmq APPEND PROPERTY IMPORTED_CONFIGURATIONS DEBUG)
set_target_properties(libzmq PROPERTIES
  IMPORTED_IMPLIB_DEBUG "${_IMPORT_PREFIX}/debug/lib/libzmq-mt-gd-4_3_2.lib"
  IMPORTED_LINK_INTERFACE_LIBRARIES_DEBUG "${_IMPORT_PREFIX}/debug/lib/libsodium.lib;ws2_32;rpcrt4;iphlpapi"
  IMPORTED_LOCATION_DEBUG "${_IMPORT_PREFIX}/debug/bin/libzmq-mt-gd-4_3_2.dll"
  )

list(APPEND _IMPORT_CHECK_TARGETS libzmq )
list(APPEND _IMPORT_CHECK_FILES_FOR_libzmq "${_IMPORT_PREFIX}/debug/lib/libzmq-mt-gd-4_3_2.lib" "${_IMPORT_PREFIX}/debug/bin/libzmq-mt-gd-4_3_2.dll" )

# Commands beyond this point should not need to know the version.
set(CMAKE_IMPORT_FILE_VERSION)

If the users use the exported CMake module in a CMake-based project like this:

find_package(ZeroMQ CONFIG REQUIRED)
target_link_libraries(main PRIVATE libzmq)

then they don't worry about the libsodium dependency (It is already recorded in the IMPORTED_LINK_INTERFACE_LIBRARIES_* property of libzmq target).

If the users use find_library instead, they have to handle the libsodium dependency manually, and all the other platform specific libraries ZeroMQ may depends on. So, I replace Findlibzmq.cmake with this one in microsoft/vcpkg#4979:

find_package(ZeroMQ CONFIG REQUIRED)

set(LIBZMQ_INCLUDE_DIRS ${ZeroMQ_INCLUDE_DIR})
set(LIBZMQ_LIBRARIES libzmq libzmq-static)

include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(
    LIBZMQ
    REQUIRED_VARS LIBZMQ_LIBRARIES LIBZMQ_INCLUDE_DIRS
)

and apply this patch to the cmake/Config.cmake.in:

diff --git a/builds/cmake/Config.cmake.in b/builds/cmake/Config.cmake.in
index 9c15f36a..e1475cd6 100644
--- a/builds/cmake/Config.cmake.in
+++ b/builds/cmake/Config.cmake.in
@@ -1,4 +1,14 @@
 @PACKAGE_INIT@

+include(CMakeFindDependencyMacro)
+
+find_dependency(ZeroMQ)
+
+if (@CZMQ_WITH_LIBCURL@ AND @LIBCURL_FOUND@)
+    find_dependency(OpenSSL)
+    find_dependency(ZLIB)
+endif ()
+
+
 include("${CMAKE_CURRENT_LIST_DIR}/@PROJECT_NAME@Targets.cmake")
 check_required_components("@PROJECT_NAME@")

I noticed the existence of #1939, and totally understand @bluca's concern. So a better solution should take both of the two cases into consideration:

find_package(ZeroMQ CONFIG QUIET)
if (ZeroMQ_FOUND)
    set(ZMQ_CMAKE_MODULE_AVAILABLE 1)
else ()
    set(ZMQ_CMAKE_MODULE_AVAILABLE 0)
    # Fall back to find_library
    ......
endif ()

and in the cmake/Config.cmake.in:

if (@ZMQ_CMAKE_MODULE_AVAILABLE@)
    find_dependency(ZeroMQ)
endif ()

czmq's another optional dependency, libcurl, also faces this problem. libcurl itself may depends on OpenSSL, zlib, and others. So:

find_library (
    LIBCURL_LIBRARIES
    NAMES curl
    HINTS ${PC_LIBCURL_LIBRARY_HINTS}
)

is not enough when libcurl is built as a static library. Besides, on Windows, the lib name of libcurl is not always libcurl.lib, so I have to take care of them this way:

find_library(
    LIBCURL_LIBRARY_DEBUG
    NAMES libcurl-d_imp libcurl-d curl-d curl
)

find_library(
    LIBCURL_LIBRARY_RELEASE
    NAMES libcurl_imp libcurl curl
)

google-cloud-cpp handles libcurl dependency this way: https://github.com/googleapis/google-cloud-cpp/pull/2407/files

So you see, a generic Find*.cmake template sounds great, but it doesn't work all the time. If we can mantain those Find*.cmake standalone, then we may refine them freely. Otherwise, we have to use a bounch of if else in zproject_cmake.gsl to handle all the special cases when a generic solution is not enough (like what I did in zeromq/zproject#1168).

If we do not want to go that far currently:

1. For *nix users, things seem not that bad. So, we do not have to change anything.
2. For Windows users with vcpkg (VS2015+ required), they may install czmq via:
   vcpkg install czmq:x86-windows
   or
   vcpkg install czmq[curl]:x86-windows
   to build czmq with libcurl.
3. For Windows users without vcpkg or with older versions of Visual Studio, they may use the deprecated Visual Studio solutions or CMake. And when they use CMake, they should handle czmq's indirect dependencies (libsodium, OpenSSL, zlib, etc.) themself or just forget about them if they don't need them.

[myd7349]

OK. Before I close this issue, I want to do two more things:

1. Add CZMQ_WITH_LIBCURL, CZMQ_WITH_LZ4, CZMQ_WITH_UUID options to give the users more control over those optional 3rd dependencies so that I don't have to use a patch here: https://github.com/microsoft/vcpkg/pull/4979/files#diff-3aedb01dfe69985b0650dc857d7b6e1a
2. Update CZMQ's install instruction for Windows users.