kroshu/kuka_drivers

ECI Mock driver not starting in Docker Container

hofermo opened this issue · 12 comments

hofermo commented

Hi there,

I am currently trying to get the ECI driver running in a Docker Container.
I am following the tutorial from the docs to start the ECI driver in a mocked test environment to control the simulated robot with ROS.

TL:DR

When trying to run ros2 launch kuka_rox_hw_interface startup.launch.py:

...
[control_node-1] [INFO] [1695046574.399726173] [resource_manager]: Loading hardware 'LBRiisy3R760' 
[control_node-1] terminate called after throwing an instance of 'pluginlib::LibraryLoadException'
[control_node-1]   what():  Failed to load library /custom_ws/install/kuka_rox_hw_interface/lib/kuka_rox_hw_interface/libkuka_rox_hw_interface.so. Make sure that you are calling the PLUGINLIB_EXPORT_CLASS macro in the library code, and that names are consistent between this macro and your XML. Error string: Could not load library dlopen error: libos-core-udp-communication.so: cannot open shared object file: No such file or directory, at ./src/shared_library.c:99
...

Setup

  • Ubuntu 20.04
  • Docker version 24.0.6
  • KUKA libs are mocked (set(MOCK_KUKA_LIBS TRUE))

Steps to reproduce

  • Copy Dockerfile below
  • Run docker build -t eci_issue .
  • Enter Docker container via docker run -it eci_issue /bin/bash (if you want a display passthrough add -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY and on the host execute xhost +)
  • Execute the command above ros2 launch kuka_rox_hw_interface startup.launch.py

Files and Outputs

This is my Dockerfile (currently using the default ros:humble entrypoint)

FROM ros:humble

RUN apt-get update &&  apt-get install -y \
    libnanopb-dev \
    ros-humble-controller-manager \
    ros-humble-moveit \
    ros-humble-moveit-visual-tools \
    ros-humble-joint-trajectory-controller \
    ros-humble-joint-state-broadcaster \
    ros-humble-joint-state-publisher \
    ros-humble-ros2-controllers \
    libprotobuf-dev \
    protobuf-compiler \
    libprotoc-dev \
    libgrpc++-dev \
    byobu \
 && rm -rf /var/lib/apt/lists/* # byobu is like tmux only for testing

RUN mkdir -p /custom_ws/src

WORKDIR /custom_ws/src

RUN git clone https://github.com/kroshu/kroshu_ros2_core.git
RUN git clone https://github.com/kroshu/ros2_kuka_drivers.git
RUN git clone https://github.com/kroshu/kuka_simulators.git

WORKDIR /custom_ws

RUN /bin/bash -c "source /opt/ros/humble/setup.bash && colcon build"
RUN /bin/bash -c "echo 'source /custom_ws/install/setup.bash' >> ~/.bashrc"

This should install all dependencies and successfully builds.
It is possible to start the rsi simulation, but when I want to start the eci startup.launch.py file it appears not to be working:

Console output when running ros2 launch kuka_rox_hw_interface startup.launch.py:

[INFO] [launch]: All log files can be found below /root/.ros/log/2023-09-18-14-16-13-840313-5c6f4854484c-53611
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [control_node-1]: process started with pid [53642]
[INFO] [robot_manager_node-2]: process started with pid [53644]
[INFO] [robot_state_publisher-3]: process started with pid [53646]
[INFO] [spawner-4]: process started with pid [53648]
[INFO] [spawner-5]: process started with pid [53650]
[INFO] [spawner-6]: process started with pid [53652]
[INFO] [spawner-7]: process started with pid [53654]
[INFO] [spawner-8]: process started with pid [53656]
[control_node-1] [INFO] [1695046574.399726173] [resource_manager]: Loading hardware 'LBRiisy3R760' 
[control_node-1] terminate called after throwing an instance of 'pluginlib::LibraryLoadException'
[control_node-1]   what():  Failed to load library /custom_ws/install/kuka_rox_hw_interface/lib/kuka_rox_hw_interface/libkuka_rox_hw_interface.so. Make sure that you are calling the PLUGINLIB_EXPORT_CLASS macro in the library code, and that names are consistent between this macro and your XML. Error string: Could not load library dlopen error: libos-core-udp-communication.so: cannot open shared object file: No such file or directory, at ./src/shared_library.c:99
[robot_manager_node-2] [INFO] [1695046574.408373075] [robot_manager]: Starting Robot Manager Node init
[robot_state_publisher-3] [INFO] [1695046574.410060310] [robot_state_publisher]: got segment base
[robot_state_publisher-3] [INFO] [1695046574.411010976] [robot_state_publisher]: got segment base_link
[robot_state_publisher-3] [INFO] [1695046574.411682061] [robot_state_publisher]: got segment base_link_inertia
[robot_state_publisher-3] [INFO] [1695046574.412352606] [robot_state_publisher]: got segment flange
[robot_state_publisher-3] [INFO] [1695046574.413030209] [robot_state_publisher]: got segment link_1
[robot_manager_node-2] [INFO] [1695046574.413413854] [robot_manager]: Control mode change requested
[robot_manager_node-2] [INFO] [1695046574.413516440] [robot_manager]: Control mode change process has started
[robot_manager_node-2] [INFO] [1695046574.413538579] [robot_manager]: Successfully changed control mode to JOINT_POSITION_CONTROL
[robot_state_publisher-3] [INFO] [1695046574.414565928] [robot_state_publisher]: got segment link_2
[robot_state_publisher-3] [INFO] [1695046574.414583618] [robot_state_publisher]: got segment link_3
[robot_state_publisher-3] [INFO] [1695046574.414595747] [robot_state_publisher]: got segment link_4
[robot_state_publisher-3] [INFO] [1695046574.414606986] [robot_state_publisher]: got segment link_5
[robot_state_publisher-3] [INFO] [1695046574.414617416] [robot_state_publisher]: got segment link_6
[robot_state_publisher-3] [INFO] [1695046574.414628271] [robot_state_publisher]: got segment tool0
[ERROR] [control_node-1]: process has died [pid 53642, exit code -6, cmd '/custom_ws/install/kroshu_ros2_core/lib/kroshu_ros2_core/control_node --ros-args --params-file /tmp/launch_params_72gydy78 --params-file /custom_ws/install/kuka_rox_hw_interface/share/kuka_rox_hw_interface/config/ros2_controller_config.yaml'].
[spawner-4] [INFO] [1695046576.717837119] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' node to exist
[spawner-5] [INFO] [1695046576.728134712] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' node to exist
[spawner-7] [INFO] [1695046576.803097720] [spawner_effort_controller]: Waiting for '/controller_manager' node to exist
[spawner-6] [INFO] [1695046577.132476945] [spawner_joint_impedance_controller]: Waiting for '/controller_manager' node to exist
[spawner-8] [INFO] [1695046577.132582531] [spawner_control_mode_handler]: Waiting for '/controller_manager' node to exist
[spawner-4] [INFO] [1695046578.745568844] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' node to exist
[spawner-5] [INFO] [1695046578.755452458] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' node to exist
[spawner-7] [INFO] [1695046578.834907075] [spawner_effort_controller]: Waiting for '/controller_manager' node to exist
[spawner-8] [INFO] [1695046579.162785457] [spawner_control_mode_handler]: Waiting for '/controller_manager' node to exist
[spawner-6] [INFO] [1695046579.164068878] [spawner_joint_impedance_controller]: Waiting for '/controller_manager' node to exist
[spawner-4] [INFO] [1695046580.781078237] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' node to exist
[spawner-5] [INFO] [1695046580.791397289] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' node to exist
[spawner-7] [INFO] [1695046580.865754205] [spawner_effort_controller]: Waiting for '/controller_manager' node to exist
[spawner-6] [INFO] [1695046581.197786033] [spawner_joint_impedance_controller]: Waiting for '/controller_manager' node to exist
[spawner-8] [INFO] [1695046581.197796166] [spawner_control_mode_handler]: Waiting for '/controller_manager' node to exist
[spawner-4] [INFO] [1695046582.815092923] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' node to exist
[spawner-5] [INFO] [1695046582.823404779] [spawner_joint_trajectory_controller]: Waiting for '/controller_manager' node to exist
[spawner-7] [INFO] [1695046582.898616650] [spawner_effort_controller]: Waiting for '/controller_manager' node to exist
[spawner-6] [INFO] [1695046583.233161777] [spawner_joint_impedance_controller]: Waiting for '/controller_manager' node to exist
[spawner-8] [INFO] [1695046583.233161784] [spawner_control_mode_handler]: Waiting for '/controller_manager' node to exist
[spawner-4] [ERROR] [1695046584.848466131] [spawner_joint_state_broadcaster]: Controller manager not available
[spawner-5] [ERROR] [1695046584.855268016] [spawner_joint_trajectory_controller]: Controller manager not available
[spawner-7] [ERROR] [1695046584.924453910] [spawner_effort_controller]: Controller manager not available
[ERROR] [spawner-4]: process has died [pid 53648, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner joint_state_broadcaster -c /controller_manager -p --inactive --ros-args'].
[ERROR] [spawner-5]: process has died [pid 53650, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner joint_trajectory_controller -c /controller_manager -p /custom_ws/install/kuka_rox_hw_interface/share/kuka_rox_hw_interface/config/joint_trajectory_controller_config.yaml --inactive --ros-args'].
[ERROR] [spawner-7]: process has died [pid 53654, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner effort_controller -c /controller_manager -p /custom_ws/install/kuka_rox_hw_interface/share/kuka_rox_hw_interface/config/effort_controller_config.yaml --inactive --ros-args'].
[spawner-8] [ERROR] [1695046585.272609936] [spawner_control_mode_handler]: Controller manager not available
[spawner-6] [ERROR] [1695046585.273111428] [spawner_joint_impedance_controller]: Controller manager not available
[ERROR] [spawner-8]: process has died [pid 53656, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner control_mode_handler -c /controller_manager -p --inactive --ros-args'].
[ERROR] [spawner-6]: process has died [pid 53652, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner joint_impedance_controller -c /controller_manager -p /custom_ws/install/kuka_rox_hw_interface/share/kuka_rox_hw_interface/config/joint_impedance_controller_config.yaml --inactive --ros-args'].
[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[robot_manager_node-2] [INFO] [1695046604.619778206] [rclcpp]: signal_handler(signum=2)
[INFO] [robot_state_publisher-3]: process has finished cleanly [pid 53646]
[INFO] [robot_manager_node-2]: process has finished cleanly [pid 53644]
[robot_state_publisher-3] [INFO] [1695046604.619803602] [rclcpp]: signal_handler(signum=2)

Output, when trying to run ros2 lifecycle set robot_manager configure:

root@d354a4fe0b00:/custom_ws# ros2 lifecycle set robot_manager configure
Transitioning failed
[robot_manager_node-2][ERROR] [robot_manager]: Could not configure hardware interface

As you can see, the control node throws an exception when Loading the Mock library for the udp communication, which probably is the point of failure here. But when we look into the CMakeLists the library should be added.

Do you also encounter this error or have an idea how to fix it?

kind regards

Moritz

Svastits commented

Hi,

We have tested this on ROS2 humble (Ubuntu 22.04), where no such issues occured. My first guess is that Ubuntu 20.04 has some incompabilities and that is the cause of this issue, as that is not the first choice for humble...

hofermo commented

I will try to spin up a VM and try it out in Ubuntu 22.04, but IMO it shouldn't matter as everything is being built inside the official ros:humble Docker container.
I will post an update if it worked 👍

hofermo commented

Update:

  • When creating the Docker Container inside Ubuntu 22.04 (freshly installed VM) the same error occures.
  • I also tried running directly inside the VM, without the abstraction of a Docker Container, on a fresh installation of ROS 2 Humble the same error occures

Is it possible, that the mock for the os-core-udp-communication library, is not properly linked / added to the executable, resulting in the control node to terminate.

hofermo commented

I found a possible solution:
When running echo $LD_LIBRARY_PATH we can see, that the kuka_rox_hw_interface is not in our library path:

/custom_ws/install/kuka_rsi_hw_interface/lib:/custom_ws/install/kuka_driver_interfaces/lib:/custom_ws/install/kroshu_ros2_core/lib:/custom_ws/install/fri/lib:/opt/ros/humble/opt/rviz_ogre_vendor/lib:/opt/ros/humble/lib/x86_64-linux-gnu:/opt/ros/humble/lib

I then manually installed the os-core-udp-communication by adding the following config to /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists.txt after line 88:

...
install(TARGETS os-core-udp-communication
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
...

Now after resourcing the setup.bash file the error disappears and the path is added to the LD_LIBRARY_PATH:

root@90b7c46baa71:/custom_ws# source install/setup.bash 
root@90b7c46baa71:/custom_ws# echo $LD_LIBRARY_PATH 
/custom_ws/install/kuka_rox_hw_interface/lib:/custom_ws/install/kuka_rsi_hw_interface/lib:/custom_ws/install/kuka_driver_interfaces/lib:/custom_ws/install/kroshu_ros2_core/lib:/custom_ws/install/fri/lib:/opt/ros/humble/opt/rviz_ogre_vendor/lib:/opt/ros/humble/lib/x86_64-linux-gnu:/opt/ros/humble/lib
hofermo commented

For anyone wondering how the now working Dockerfile looks like, here it is:

FROM ros:humble

RUN apt-get update &&  apt-get install -y \
    libnanopb-dev \
    ros-humble-controller-manager \
    ros-humble-moveit \
    ros-humble-moveit-visual-tools \
    ros-humble-joint-trajectory-controller \
    ros-humble-joint-state-broadcaster \
    ros-humble-joint-state-publisher \
    ros-humble-ros2-controllers \
    libprotobuf-dev \
    protobuf-compiler \
    libprotoc-dev \
    libgrpc++-dev \
    byobu \
 && rm -rf /var/lib/apt/lists/* # byobu is like tmux only for testing

RUN mkdir -p /custom_ws/src

WORKDIR /custom_ws/src

RUN git clone https://github.com/kroshu/kroshu_ros2_core.git
RUN git clone https://github.com/kroshu/ros2_kuka_drivers.git
RUN git clone https://github.com/kroshu/kuka_simulators.git
RUN git clone https://github.com/kroshu/ros2_kuka_controllers.git

WORKDIR /custom_ws

# Custom fix for now
RUN cp /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists.txt /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists_old.txt
RUN awk 'NR==88{print "install(TARGETS os-core-udp-communication LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})"}1' /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists_old.txt > /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists.txt


RUN /bin/bash -c "source /opt/ros/humble/setup.bash && colcon build"
RUN /bin/bash -c "echo 'source /custom_ws/install/setup.bash' >> ~/.bashrc"
...

If this issue is fixed in the future, it shouldn't be necessarry to need the two lines below # Custom fix for now

Svastits commented

Thanks for investigating this issue :)
We still could not reproduce it on VM-s or native Linux computers, but indeed the kuka_rox_hw_interface is missing from the $LD_LIBRARY_PATH. I fixed this in this branch by exporting the library in cmake:
https://github.com/kroshu/ros2_kuka_drivers/tree/fix/ld_library_path

Building your initial docker file I could reproduce the issue, using this branch seemed to fix it. (However you will also need to clone the kuka_controllers repo to get rid of the errors about being unable to load the custom controllers found there)

Could you verify that this solved the issue on your side too?

hofermo commented

You are right, the kuka_rox_hw_interface is now in $LD_LIBRARY_PATH, but the same error persists, as the library os-core-udp-communication is not in the kuka_rox_hw_interface folder:

root@ros_test:/custom_ws# echo $LD_LIBRARY_PATH 
/custom_ws/install/kuka_rox_hw_interface/lib:/custom_ws/install/kuka_rsi_hw_interface/lib:/custom_ws/install/kuka_controllers/lib:/custom_ws/install/kuka_driver_interfaces/lib:/custom_ws/install/kroshu_ros2_core/lib:/custom_ws/install/fri/lib:/custom_ws/install/effort_controllers/lib:/opt/ros/humble/opt/rviz_ogre_vendor/lib:/opt/ros/humble/lib/x86_64-linux-gnu:/opt/ros/humble/lib

root@ros_test:/custom_ws# ls /custom_ws/install/kuka_rox_hw_interface/lib/kuka_rox_hw_interface/
libkuka_rox_hw_interface.so  robot_manager_node

root@ros_test:/custom_ws# ls /custom_ws/install/kuka_rox_hw_interface/lib/
kuka_rox_hw_interface

Maybe you can't reproduce the error, because the os-core-udp-communication library is already in your LD_LIBRARY_PATH somewhere else?
It only works on my side if I add an install of the os-core-udp-communictaion like before.

Here is a Dockerfile using the fix/ld_library_path branch together with the kuka_controllers repo

FROM ros:humble

RUN apt-get update &&  apt-get install -y \
    libnanopb-dev \
    ros-humble-controller-manager \
    ros-humble-moveit \
    ros-humble-moveit-visual-tools \
    ros-humble-joint-trajectory-controller \
    ros-humble-joint-state-broadcaster \
    ros-humble-joint-state-publisher \
    ros-humble-ros2-controllers \
    libprotobuf-dev \
    protobuf-compiler \
    libprotoc-dev \
    libgrpc++-dev \
 && rm -rf /var/lib/apt/lists/* 

RUN mkdir -p /custom_ws/src

WORKDIR /custom_ws/src

RUN git clone https://github.com/kroshu/kroshu_ros2_core.git
RUN git clone https://github.com/kroshu/ros2_kuka_drivers.git -b fix/ld_library_path
RUN git clone https://github.com/kroshu/kuka_simulators.git
RUN git clone https://github.com/kroshu/ros2_kuka_controllers.git

WORKDIR /custom_ws

# Custom fix for now
#RUN cp /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists.txt /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists_old.txt
#RUN awk 'NR==88{print "install(TARGETS os-core-udp-communication LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} PUBLIC_HEADER DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})"}1' /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists_old.txt > /custom_ws/src/ros2_kuka_drivers/kuka_rox_hardware_interface/CMakeLists.txt

RUN /bin/bash -c "source /opt/ros/humble/setup.bash && colcon build"
RUN /bin/bash -c "echo 'source /custom_ws/install/setup.bash' >> ~/.bashrc"

RUN /bin/bash -c "echo 'source /opt/ros/humble/setup.bash && source  /custom_ws/install/setup.bash && ros2 launch kuka_rox_hw_interface startup.launch.py' > /custom_ws/startup.bash"
RUN chmod +x /custom_ws/startup.bash

ENTRYPOINT ["/bin/bash", "/custom_ws/startup.bash"]

Sorry for the troubles 😁

PS: I also tested it on a virtual machine and there was the same error :(

Svastits commented

Yesterday I used your Dockerfile and the fix seemed to have resolved the issue, but I will look at it again then :)

I think I have the original (non-mock) version of the udp lib installed on my PC, that could indeed explain why the issue was not there for me

  • 👍1
Svastits commented

I think I found the reason for not being able to reproduce this bug previously: I always use the --symlink-install flag for colcon build, which installs a symbolic link of the .so file located in the build folder. This .so has the udp lib right next to it so the library loader can find it.

So the quickest fix would be to modify your docker file to use --symlink-install, but I will also modify cmake soon to fix the "normal" build

  • ❤️1
hofermo commented

So the quickest fix would be to modify your docker file to use --symlink-install, but I will also modify cmake soon to fix the "normal" build

This seams to fix it, thank you for investigating 😄
I adjusted the Dockerfile (see below if anyone runs into the same problem).

FROM ros:humble

RUN apt-get update &&  apt-get install -y \
    libnanopb-dev \
    ros-humble-controller-manager \
    ros-humble-moveit \
    ros-humble-moveit-visual-tools \
    ros-humble-joint-trajectory-controller \
    ros-humble-joint-state-broadcaster \
    ros-humble-joint-state-publisher \
    ros-humble-ros2-controllers \
    libprotobuf-dev \
    protobuf-compiler \
    libprotoc-dev \
    libgrpc++-dev \
 && rm -rf /var/lib/apt/lists/* 

RUN mkdir -p /custom_ws/src

WORKDIR /custom_ws/src

RUN git clone https://github.com/kroshu/kroshu_ros2_core.git
RUN git clone https://github.com/kroshu/ros2_kuka_drivers.git -b fix/ld_library_path
RUN git clone https://github.com/kroshu/kuka_simulators.git
RUN git clone https://github.com/kroshu/ros2_kuka_controllers.git

WORKDIR /custom_ws

RUN /bin/bash -c "source /opt/ros/humble/setup.bash && colcon build --symlink-install"
RUN /bin/bash -c "echo 'source /custom_ws/install/setup.bash' >> ~/.bashrc"

RUN /bin/bash -c "echo 'source /opt/ros/humble/setup.bash && source  /custom_ws/install/setup.bash && ros2 launch kuka_rox_hw_interface startup.launch.py' > /custom_ws/startup.bash"
RUN chmod +x /custom_ws/startup.bash

ENTRYPOINT ["/bin/bash", "/custom_ws/startup.bash"]

Quick note, if fix/ld_library_path is merged into the main branch in the future, just remove the -b fix/ld_library_path flag on git clone

Svastits commented

I modified the Cmakelists to install the udp library in mock mode.
The PR closed the issue, I will reopen it, please close it if it is fixed on your side too

  • 👍1
hofermo commented

Looks good to me, thank you so much for the support 😄

  • 👍1