morganstanley/modern-cpp-kafka

Make CMake ExternalProject-friendly

jm4R opened this issue · 7 comments

jm4R commented

The good practice when writing modern CMake is make it friendly for ExternalProject and therefore FetchContent
Typical usage:

FetchContent_Declare(cppkafka
    URL      https://github.com/morganstanley/modern-cpp-kafka/archive/refs/tags/v2021.12.08.zip
    URL_HASH SHA256=3eef58d076ccaebd75670826b7ff8866b619e2085f4b59e96dc235695d3bc166
)
FetchContent_MakeAvailable(cppkafka)

target_link_libraries(mytarget PRIVATE
    fmt::fmt
    some_other_lib::core
    cppkafka::core
)

When I tried, I meet following issues:

  • The rapidjson dependency was added to my project as it is required by non-optional tests/unit/CMakeLists.txt module. The common practice is to disable tests by default when PROJECT_IS_TOP_LEVEL variable is false. This requires newer CMake version but there is workaround for older.
  • Includes inside the library's headers doesn't work as CMake installs headers path as SYSTEM (target_include_directories(${PROJECT_NAME} SYSTEM INTERFACE ${LIBRDKAFKA_INCLUDE_DIR})) but code uses quotes syntax. Elegant fix would be to change quotes to angle brackets.
  • Minor issue is that target modern-cpp-kafka-api has no elegant alias with modular syntax like cppkafka::core. But we can live without it.
kenneth-jia commented

Thank you, @jm4R ! Yes, definitely we should move to "modern CMake", it's in my backlog now. 😉

SuperElectron commented

docker run -it ubuntu:20.04 /bin/bash

root@043e2bacf4c7:/# apt-get update -y -qq && apt-get install -y -qq wget unzip make cmake g++ pkg-config

root@043e2bacf4c7:/# mkdir -p /start && cd /start &&
wget https://github.com/edenhill/librdkafka/archive/refs/tags/v1.9.0.zip &&
unzip v1.9.0.zip && rm -rf v1.9.0.zip

root@043e2bacf4c7:/# cd /start/librdkafka-1.9.0 && ./configure --install-deps --source-deps-only;

root@043e2bacf4c7:/# cd /start/librdkafka-1.9.0 && make
root@043e2bacf4c7:/# cd /start/librdkafka-1.9.0 && make install

root@043e2bacf4c7:/# apt-get install -y libgtest-dev curl zip

root@043e2bacf4c7:/# cd /start &&
wget https://github.com/Tencent/rapidjson/archive/refs/tags/v1.1.0.zip &&
unzip v1.1.0.zip && rm -rf v1.1.0.zip

root@043e2bacf4c7:/# cd /start/rapidjson-1.1.0 && cmake .
root@043e2bacf4c7:/# cd /start/rapidjson-1.1.0 && make install

root@043e2bacf4c7:/# apt-get update -y -qq && apt-get install -y clang-tidy doxygen libboost-all-dev rapidjson-dev
root@043e2bacf4c7:/# cd /start &&
wget https://github.com/morganstanley/modern-cpp-kafka/archive/refs/tags/v2022.06.15.zip &&
unzip v2022.06.15.zip && rm -rf v2022.06.15.zip;

root@043e2bacf4c7:/# cd /start/modern-cpp-kafka-2022.06.15 && mkdir -p build && cd build

root@043e2bacf4c7:/start/modern-cpp-kafka-2022.06.15/build# cmake ..
-- librdkafka include directory: /usr/local/include
-- librdkafka library directory: /usr/local/lib
-- pthread library: /usr/lib/x86_64-linux-gnu/libpthread.so
-- With NO clang-tidy build option
-- boost include directory: /usr/include
-- boost library directory:
-- googletest root directory: /usr/local
CMake Error at tests/unit/CMakeLists.txt:9 (find_package):
By not providing "Findrapidjson.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"rapidjson", but CMake did not find one.

Could not find a package configuration file provided by "rapidjson" with
any of the following names:

rapidjsonConfig.cmake
rapidjson-config.cmake

Add the installation prefix of "rapidjson" to CMAKE_PREFIX_PATH or set
"rapidjson_DIR" to a directory containing one of the above files. If
"rapidjson" provides a separate development package or SDK, be sure it has
been installed.

-- Configuring incomplete, errors occurred!
See also "/start/modern-cpp-kafka-2022.06.15/CMakeFiles/CMakeOutput.log".

I have been stuck here for a while. Any tips?

root@043e2bacf4c7:/start/modern-cpp-kafka-2022.06.15/build# g++ --version
g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@043e2bacf4c7:/start/modern-cpp-kafka-2022.06.15/build# cmake --version
cmake version 3.16.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

kenneth-jia commented

Hi, @SuperElectron
Thank you for pointing it out!
Previously, the CMake project could not find the "installed" rapidjson package, and it has just been fixed with #156
Please have a try with the pre-release https://github.com/morganstanley/modern-cpp-kafka/releases/tag/v2022.08.01

  • ❤️1
SuperElectron commented

rapidjson works. I just moved back to this development and ran into another problem.

I have librdkafka version 1.9.0, cmake compiles, but i get this error when trying to run the C++ file

./run: symbol lookup error: ./run: undefined symbol: rd_kafka_incremental_unassign

I have my own CMakeLists.txt, but because the examples you provide build multiple files it is hard to understand how to use the "header only" in my own project.

  1. Is there an example of using this in a standalone project?
  2. Do you know why the symbol lookup error for "rd_kafka_incremental_unassign" happens when I try and run a program?
  3. Are there required steps to using librdkafka with cpp-modern-kafka in a CMake file that I can follow?

thanks :)

SuperElectron commented

The previous errors was building librdkafka with ./configure and some args ...

I am trying to run the file below and get errors building ...

`docker run -it openvino/ubuntu20_data_dev /bin/bash -v ./install_messaging.sh:/start/scripts/install_messaging.sh -w /start/scripts

root@010c67042f92:/start/scripts# ./install_messaging.sh

...

cd "${WORKDIR}/modern-cpp-kafka-${KAFKA_VERSION_TAG}/build" && cmake .. "-DCMAKE_TOOLCHAIN_FILE=/start/vcpkg/scripts/buildsystems/vcpkg.cmake"
CMake Error at CMakeLists.txt:53 (message):
Could not find headers: librdkafka!

`#!/bin/bash

Install cpp-modern-kafka and its deps

Example usage: ./install_messaging.sh

export NAME="[install_messaging.sh]: "

set -eE -o functrace
failure() {
local lineno=$1
local msg=$2
echo "${NAME} Failed at $lineno: $msg"
}
trap '${NAME} failure ${LINENO} "$BASH_COMMAND"' ERR

Script start

echo "${NAME} STARTING "
export WORKDIR=/start

echo "${NAME} Install 'librdkafka' with vcpkg "
cd "${WORKDIR}" && git clone https://github.com/Microsoft/vcpkg.git
cd "${WORKDIR}/vcpkg"; ./bootstrap-vcpkg.sh;
cd "${WORKDIR}/vcpkg"; ./vcpkg integrate install;
cd "${WORKDIR}/vcpkg"; ./vcpkg install librdkafka;

echo "${NAME} Install 'modern-cpp-kafka' from source "
export KAFKA_VERSION_TAG=2022.08.01
apt-get update -y -qq && apt-get install -y clang-tidy doxygen libboost-all-dev libgtest-dev curl zip rapidjson-dev
cd ${WORKDIR} &&
wget "https://github.com/morganstanley/modern-cpp-kafka/archive/refs/tags/v${KAFKA_VERSION_TAG}.zip" &&
unzip "v${KAFKA_VERSION_TAG}.zip" && rm -rf "v${KAFKA_VERSION_TAG}.zip";
cd "${WORKDIR}/modern-cpp-kafka-${KAFKA_VERSION_TAG}" && mkdir -p build;
cd "${WORKDIR}/modern-cpp-kafka-${KAFKA_VERSION_TAG}/build" && cmake .. "-DCMAKE_TOOLCHAIN_FILE=/start/vcpkg/scripts/buildsystems/vcpkg.cmake"
cd "${WORKDIR}/modern-cpp-kafka-${KAFKA_VERSION_TAG}/build" && make -j
cd "${WORKDIR}/modern-cpp-kafka-${KAFKA_VERSION_TAG}/build" && make install

echo "${NAME} FINISHED "
`

How do I build your project?

kenneth-jia commented

Hi, @SuperElectron

  • 👍1
SuperElectron commented

this worked. thank you

  • 👍1