Cross-compilation Pepper robot
Closed this issue · 9 comments
Hello,
I am trying to cross-compile this lib for Pepper. I am building everything from source using python3.8 and boost 1.71 in a docker container, here is my fork:
https://github.com/Maelic/libqi-python
I am able to compile but when I try to execute on the robot I get:
nao@88ea9a3a7e80 ~ $ python
Python 3.8.12 (default, Mar 9 2022, 13:49:35)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import qi
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/nao/.local/bin/libqi_python/lib/python3.8/site-packages/qi/__init__.py", line 16, in <module>
from .qi_python \
ImportError: /home/nao/.local/bin/libqi_python/lib/python3.8/site-packages/qi/qi_python.so: undefined symbol: _ZN2qi10Translator16setDefaultDomainERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
>>>
Do you have any idea where this can come from?
Hello @Maelic,
Nice work on compiling everything :) we're aware it's not a simple process, and we're looking into improving it.
The error you're experiencing is probably caused by files that were not compiled in libqi (qi::Translator files to be exact). It seems like an error in our code to try to use these symbols when their compilation was disabled.
Can you try activating BOOST_LOCALE support in libqi if not done already and ensure you have it in your version of boost ? This can be done by passing -DWITH_BOOST_LOCALE=TRUE at configure time.
Hi @nyibbang thank you for your help,
I just tried to compile libqi with -DWITH_BOOST_LOCALE=TRUE but it didn't change anything. Here is my fork of libqi where I changed the supported version of boost to 1.71:
https://github.com/Maelic/libqi
My way of cross-compiling everything is by using the docker image of pepper from here https://hub.docker.com/r/awesomebytes/pepper_2.5.5.5 and then in a dockerfile (where I have boost 1.71 and python3):
RUN mkdir -p /home/nao/.local &&\
cd /home/nao/.local &&\
git clone https://github.com/Maelic/libqi.git && \
cd libqi && \
mkdir build && cd build && \
cmake .. -DQI_WITH_TESTS=OFF -DCMAKE_PREFIX_PATH=/home/nao/.local/qibuild/cmake/qibuild -DWITH_BOOST_LOCALE=TRUE && \
make -j8 && \
make install DESTDIR=/home/nao/.local/bin/libqi
RUN mkdir -p /home/nao/.local &&\
cd /home/nao/.local &&\
git clone https://github.com/Maelic/libqi-python.git && cd libqi-python && \
mkdir build && cd build && \
cmake .. -DQI_WITH_TESTS=OFF -DCMAKE_PREFIX_PATH="/home/nao/.local/bin/libqi/share/cmake/qi;/data/home/nao/.local/qibuild/cmake/qibuild/modules;/home/nao/.local/qibuild/cmake/qibuild;/home/nao/.local/bin/libqi/share/cmake/qimodule" -DCMAKE_INSTALL_PREFIX=/home/nao/.local/bin/libqi_python && \
cmake --build . && \
cmake --install .
Okay, I tried locally on my machine (Ubuntu 20.04, with boost from canonical repositories) and I could reproduce the same issue by disabling WITH_BOOST_LOCALE during my libqi build. I could not reproduce it without explicitly specifying the option to false though.
Can you check that src/translator.cpp from libqi was built correctly ? and then that the symbols are in the library : objdump -tw libqi.so | c++filt | grep Translator::setDefaultDomain should return something. The WITH_BOOST_LOCALE option should force CMake to build it but for some reason it doesn't seem to be working in your case.
Here is the output:
nao@364a25303471 ~/.local/bin/libqi/lib $ objdump -tw libqi.so | c++filt | grep Translator::setDefaultDomain
003753d2 g F .text 00000027 qi::Translator::setDefaultDomain(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
I tried with and without -DWITH_BOOST_LOCALE=TRUE and each time src/translator.cpp is built correctly.
I looked online today and actually it may be an error related to the dual ABI from gcc https://gcc.gnu.org/onlinedocs/gcc-11.2.0/libstdc++/manual/manual/using_dual_abi.html with the std::__cxx11::basic_string symbol.
But that doesn't really make sense because I'm compiling both libqi and libqi_python with the same version of c++ and the same version of the ABI (gcc 10.3 that support c++11)...
Here are my settings for building libqi:
Cloning into 'libqi'...
-- The C compiler identification is GNU 10.3.0
-- The CXX compiler identification is GNU 10.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /tmp/gentoo/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /tmp/gentoo/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using qibuild 3.16
-- Looking for _SC_HOST_NAME_MAX
-- Looking for _SC_HOST_NAME_MAX - found
-- Library: qi
-- Binary: qipath_example
-- Binary: sharedlibrary_example
-- Binary: log_example
-- Binary: buffer_example
-- Binary: qiconvloc_example
-- Using qibuild 3.16
-- Binary: test_translate
-- Using qibuild 3.16
-- Module: foo
-- Binary: footest
-- Library: qitestutils
-- Library: testsession
And the same for libqi_python:
Cloning into 'libqi-python'...
-- The C compiler identification is GNU 10.3.0
-- The CXX compiler identification is GNU 10.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /tmp/gentoo/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /tmp/gentoo/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using qibuild 3.16
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found OpenSSL: /tmp/gentoo/usr/lib/libcrypto.so (found version "1.1.1k")
-- Found Python: /tmp/gentoo/usr/bin/python3.8 (found suitable exact version "3.8.12") found components: Development Interpreter Development.Module Development.Embed
-- Found Boost: /tmp/gentoo/usr/include (found suitable exact version "1.71.0") found components: atomic date_time thread chrono filesystem locale regex program_options random
-- Found the following ICU libraries:
-- i18n (required)
-- uc (required)
-- data (required)
-- Found ICU: /tmp/gentoo/usr/include (found version "70.1")
-- Found PythonInterp: /tmp/gentoo/usr/bin/python (found version "3.8.12")
-- LibQi: expected version is "2.0.0"
-- LibQi: found version "2.0.0" from file '/home/nao/.local/libqi/package.xml'
-- tests: tests are disabled
But that doesn't really make sense because I'm compiling both libqi and libqi_python with the same version of c++ and the same version of the ABI (gcc 10.3 that support c++11)...
Yes, I thought of it but i'ts unlikely as you said. Could you build in verbose mode (cmake --build ... --verbose or VERBOSE=1 make ...) and paste the output ?
One another possible cause would be the libqi/libqi-python that is on the docker image you're using, which is the one that could be loaded when you use import qi (it loads your qi_python.so which then tries to load the libqi.so from /opt/aldebaran/lib/libqi.so). It should not happen, because the qi_python.so should have correct RUNPATH set, but maybe there is a LD_LIBRARY_PATH lurking around messing things up.
This is the output for libqi translator.cpp :
[ 25%] Building CXX object CMakeFiles/qi.dir/src/translator.cpp.o
/tmp/gentoo/usr/bin/c++ -DBOOST_ASIO_DISABLE_STD_CHRONO -DBOOST_FILESYSTEM_VERSION=3 -Dqi_EXPORTS
-I/home/nao/.local/libqi/build/include -I/home/nao/.local/libqi -Wall -Wextra -std=gnu++11 -g -fPIC -Wall -Wno-unused-parameter
-Werror=return-type -fvisibility=hidden -fno-strict-aliasing -DWITH_BOOST_LOCALE -fPIC -MD -MT CMakeFiles/qi.dir/src/translator.cpp.o -MF CMakeFiles/qi.dir/src/translator.cpp.o.d -o CMakeFiles/qi.dir/src/translator.cpp.o -c
/home/nao/.local/libqi/src/translator.cpp
And the build of libqi.so :
[ 76%] Linking CXX shared library sdk/lib/libqi.so
/data/home/nao/gentoo/usr/bin/cmake -E cmake_link_script CMakeFiles/qi.dir/link.txt --verbose=1
/tmp/gentoo/usr/bin/c++ -fPIC -Wall -Wextra -std=gnu++11 -g -shared -Wl,-soname,libqi.so -o sdk/lib/libqi.so
CMakeFiles/qi.dir/src/dlfcn.cpp.o CMakeFiles/qi.dir/src/path.cpp.o CMakeFiles/qi.dir/src/application.cpp.o
CMakeFiles/qi.dir/src/buffer.cpp.o CMakeFiles/qi.dir/src/bufferreader.cpp.o CMakeFiles/qi.dir/src/clock.cpp.o
CMakeFiles/qi.dir/src/future.cpp.o CMakeFiles/qi.dir/src/log.cpp.o CMakeFiles/qi.dir/src/consoleloghandler.cpp.o
CMakeFiles/qi.dir/src/fileloghandler.cpp.o CMakeFiles/qi.dir/src/csvloghandler.cpp.o
CMakeFiles/qi.dir/src/headfileloghandler.cpp.o CMakeFiles/qi.dir/src/tailfileloghandler.cpp.o CMakeFiles/qi.dir/src/locale-
light.cpp.o CMakeFiles/qi.dir/src/os.cpp.o CMakeFiles/qi.dir/src/path_conf.cpp.o CMakeFiles/qi.dir/src/periodictask.cpp.o
CMakeFiles/qi.dir/src/print.cpp.o CMakeFiles/qi.dir/src/utils.cpp.o CMakeFiles/qi.dir/src/eventloop.cpp.o
CMakeFiles/qi.dir/src/sdklayout-boost.cpp.o CMakeFiles/qi.dir/src/version.cpp.o CMakeFiles/qi.dir/src/iocolor.cpp.o
CMakeFiles/qi.dir/src/strand.cpp.o CMakeFiles/qi.dir/src/ptruid.cpp.o CMakeFiles/qi.dir/src/os_posix.cpp.o
CMakeFiles/qi.dir/src/os_debugger_posix.cpp.o CMakeFiles/qi.dir/src/os_launch_posix.cpp.o
CMakeFiles/qi.dir/src/translator.cpp.o CMakeFiles/qi.dir/src/type/binarycodec.cpp.o
CMakeFiles/qi.dir/src/type/dynamicobject.cpp.o CMakeFiles/qi.dir/src/type/dynamicobjectbuilder.cpp.o
CMakeFiles/qi.dir/src/type/anyfunction.cpp.o CMakeFiles/qi.dir/src/type/anyreference.cpp.o
CMakeFiles/qi.dir/src/type/anyvalue.cpp.o CMakeFiles/qi.dir/src/type/anyobject.cpp.o
CMakeFiles/qi.dir/src/type/genericobject.cpp.o CMakeFiles/qi.dir/src/type/jsondecoder.cpp.o
CMakeFiles/qi.dir/src/type/jsonencoder.cpp.o CMakeFiles/qi.dir/src/type/manageable.cpp.o
CMakeFiles/qi.dir/src/type/metamethod.cpp.o CMakeFiles/qi.dir/src/type/metaproperty.cpp.o
CMakeFiles/qi.dir/src/type/metasignal.cpp.o CMakeFiles/qi.dir/src/type/metasignal_p.cpp.o
CMakeFiles/qi.dir/src/type/metaobject.cpp.o CMakeFiles/qi.dir/src/type/anymodule.cpp.o
CMakeFiles/qi.dir/src/type/objecttypebuilder.cpp.o CMakeFiles/qi.dir/src/type/signal.cpp.o
CMakeFiles/qi.dir/src/type/signalspy.cpp.o CMakeFiles/qi.dir/src/type/signatureconvertor.cpp.o
CMakeFiles/qi.dir/src/type/staticobjecttype.cpp.o CMakeFiles/qi.dir/src/type/typeinterface.cpp.o
CMakeFiles/qi.dir/src/type/structtypeinterface.cpp.o CMakeFiles/qi.dir/src/type/type.cpp.o
CMakeFiles/qi.dir/src/type/signature.cpp.o CMakeFiles/qi.dir/src/type/traceanalyzer.cpp.o
CMakeFiles/qi.dir/src/messaging/applicationsession_internal.cpp.o CMakeFiles/qi.dir/src/messaging/applicationsession.cpp.o
CMakeFiles/qi.dir/src/messaging/authprovider.cpp.o CMakeFiles/qi.dir/src/messaging/boundobject.cpp.o
CMakeFiles/qi.dir/src/messaging/clientauthenticator.cpp.o CMakeFiles/qi.dir/src/messaging/gateway.cpp.o
CMakeFiles/qi.dir/src/messaging/message.cpp.o CMakeFiles/qi.dir/src/messaging/messagedispatcher.cpp.o
CMakeFiles/qi.dir/src/messaging/objecthost.cpp.o CMakeFiles/qi.dir/src/messaging/objectregistrar.cpp.o
CMakeFiles/qi.dir/src/messaging/remoteobject.cpp.o CMakeFiles/qi.dir/src/messaging/servicedirectory.cpp.o
CMakeFiles/qi.dir/src/messaging/servicedirectoryclient.cpp.o CMakeFiles/qi.dir/src/messaging/servicedirectoryproxy.cpp.o
CMakeFiles/qi.dir/src/messaging/serviceinfo.cpp.o CMakeFiles/qi.dir/src/messaging/session.cpp.o
CMakeFiles/qi.dir/src/messaging/sessionservice.cpp.o CMakeFiles/qi.dir/src/messaging/sessionservices.cpp.o
CMakeFiles/qi.dir/src/messaging/server.cpp.o CMakeFiles/qi.dir/src/messaging/streamcontext.cpp.o
CMakeFiles/qi.dir/src/messaging/transportserver.cpp.o CMakeFiles/qi.dir/src/messaging/transportserverasio_p.cpp.o
CMakeFiles/qi.dir/src/messaging/messagesocket.cpp.o CMakeFiles/qi.dir/src/messaging/transportsocketcache.cpp.o
CMakeFiles/qi.dir/src/messaging/tcpmessagesocket.cpp.o CMakeFiles/qi.dir/src/messaging/uri.cpp.o
CMakeFiles/qi.dir/src/messaging/url.cpp.o CMakeFiles/qi.dir/src/registration.cpp.o
CMakeFiles/qi.dir/src/perf/dataperfsuite.cpp.o CMakeFiles/qi.dir/src/perf/dataperf.cpp.o
CMakeFiles/qi.dir/src/perf/measure.cpp.o -Wl,-rpath,:::::::: -lboost_atomic-mt -lboost_date_time-mt -lboost_thread-mt -lpthread -lboost_chrono-mt -lboost_date_time-mt -lboost_atomic-mt -lpthread -lboost_chrono-mt -lboost_filesystem-mt -lboost_system-mt -lboost_program_options-mt -lboost_random-mt -lboost_regex-mt -lssl -lcrypto -ldl -lboost_locale-mt -lboost_thread-mt -lpthread -lboost_chrono-mt -lboost_date_time-mt -lboost_atomic-mt -lpthread -ldl -lrt -lboost_filesystem-mt -lboost_system-mt -lboost_program_options-mt -lboost_random-mt -lboost_regex-mt -lssl -lcrypto -lboost_locale-mt -lrt
make[2]: Leaving directory '/data/home/nao/.local/libqi/build'
Now the same for libqi_python pytranslator.cpp :
[ 51%] Building CXX object CMakeFiles/qi_python_objects.dir/src/pytranslator.cpp.o
/tmp/gentoo/usr/bin/c++ -I/home/nao/.local/libqi-python -I/home/nao/.local/libqi-python/build/_deps/pybind11-src/include -
I/tmp/gentoo/usr/include/python3.8 -isystem **/home/nao/.local/libqi -isystem /home/nao/.local/libqi/compat -isystem
/home/nao/.local/libqi/build/include** -g -fPIC -fvisibility=hidden -Wall -Wextra -MD -MT
CMakeFiles/qi_python_objects.dir/src/pytranslator.cpp.o -MF CMakeFiles/qi_python_objects.dir/src/pytranslator.cpp.o.d -o
CMakeFiles/qi_python_objects.dir/src/pytranslator.cpp.o -c /home/nao/.local/libqi-python/src/pytranslator.cpp
And qi_python.so :
[ 62%] Linking CXX shared module qi/qi_python.so
/data/home/nao/gentoo/usr/bin/cmake -E cmake_link_script CMakeFiles/qi_python.dir/link.txt --verbose=1
/tmp/gentoo/usr/bin/c++ -fPIC -g -shared -o qi/qi_python.so CMakeFiles/qi_python.dir/src/module.cpp.o
CMakeFiles/qi_python_objects.dir/src/pyapplication.cpp.o CMakeFiles/qi_python_objects.dir/src/pyasync.cpp.o
CMakeFiles/qi_python_objects.dir/src/pyclock.cpp.o CMakeFiles/qi_python_objects.dir/src/pyexport.cpp.o
CMakeFiles/qi_python_objects.dir/src/pyfuture.cpp.o CMakeFiles/qi_python_objects.dir/src/pylog.cpp.o
CMakeFiles/qi_python_objects.dir/src/pymodule.cpp.o CMakeFiles/qi_python_objects.dir/src/pyobject.cpp.o
CMakeFiles/qi_python_objects.dir/src/pypath.cpp.o CMakeFiles/qi_python_objects.dir/src/pyproperty.cpp.o
CMakeFiles/qi_python_objects.dir/src/pysession.cpp.o CMakeFiles/qi_python_objects.dir/src/pysignal.cpp.o
CMakeFiles/qi_python_objects.dir/src/pystrand.cpp.o CMakeFiles/qi_python_objects.dir/src/pytranslator.cpp.o
CMakeFiles/qi_python_objects.dir/src/pytypes.cpp.o -Wl,-rpath,/tmp/gentoo/usr/lib:/home/nao/.local/libqi/build/sdk/lib:
/home/nao/.local/libqi/build/sdk/lib/libqi.so /tmp/gentoo/usr/lib/libboost_atomic-mt.so /tmp/gentoo/usr/lib/libboost_date_time-
mt.so /tmp/gentoo/usr/lib/libboost_thread-mt.so -lpthread /tmp/gentoo/usr/lib/libboost_chrono-mt.so
/tmp/gentoo/usr/lib/libboost_filesystem-mt.so /tmp/gentoo/usr/lib/libboost_locale-mt.so /tmp/gentoo/usr/lib/libboost_regex-mt.so
/tmp/gentoo/usr/lib/libboost_program_options-mt.so /tmp/gentoo/usr/lib/libboost_random-mt.so
/tmp/gentoo/usr/lib/libboost_system-mt.so /tmp/gentoo/usr/lib/libssl.so /tmp/gentoo/usr/lib/libcrypto.so
/tmp/gentoo/usr/lib/libboost_atomic-mt.so /tmp/gentoo/usr/lib/libboost_date_time-mt.so /tmp/gentoo/usr/lib/libboost_thread-
mt.so -lpthread /tmp/gentoo/usr/lib/libboost_chrono-mt.so /tmp/gentoo/usr/lib/libboost_filesystem-mt.so
/tmp/gentoo/usr/lib/libboost_locale-mt.so /tmp/gentoo/usr/lib/libboost_regex-mt.so
/tmp/gentoo/usr/lib/libboost_program_options-mt.so /tmp/gentoo/usr/lib/libboost_random-mt.so
/tmp/gentoo/usr/lib/libboost_system-mt.so /tmp/gentoo/usr/lib/libssl.so /tmp/gentoo/usr/lib/libcrypto.so
/tmp/gentoo/usr/lib/libdl.so /tmp/gentoo/usr/lib/librt.so
gmake[2]: Leaving directory '/data/home/nao/.local/libqi-python/build'
[ 62%] Built target qi_python
It looks like it's using the right version of libqi.so
Okay, I finally found the issue, it is as you said related to the LD_LIBRARY_PATH:
Because I also installed ROS Noetic on the robot, when I log, I automatically source opt/ros/noetic/setup.bash and this script is adding /opt/ros/noetic/lib to the LD_LIBRARY_PATH. Then, for an unknown reason qi_python.so is not finding the right libqi.so file, so I have to reset the LD_LIBRARY_PATH.
Interesting but this is working fine:
export LD_LIBRARY_PATH=/home/nao/.local/bin/libqi/lib:/tmp/gentoo/opt/ros/noetic/lib
While this is not working, and lead to the undefined symbol error:
export LD_LIBRARY_PATH=/tmp/gentoo/opt/ros/noetic/lib:/home/nao/.local/bin/libqi/lib
Thank you for the help, I guess you can close this issue now. I can share my build of ROS Noetic + libqi with python3 for pepper if you're interested.
Interesting but this is working fine:
export LD_LIBRARY_PATH=/home/nao/.local/bin/libqi/lib:/tmp/gentoo/opt/ros/noetic/lib
Yes, this is because LD_LIBRARY_PATH sets an order for searching libraries (if the path appears first in LD_LIBRARY_PATH, then it is searched first), and it has priority over the RUNPATH set on libraries and executables.
Glad to hear that you solved your issue :) I'm closing the issue.