ZED-X data timestamps not updated in docker image
Closed this issue · 5 comments
Preliminary Checks
- This issue is not a duplicate. Before opening a new issue, please search existing issues.
- This issue is not a question, feature request, or anything other than a bug report directly related to this project.
Description
I am using:
- ZED-X + ZED-XM
- on Jetson AGX Orin
- from within docker container
Cameras work and output data but with same timestamp all the time.
This breaks other parts of the system that are depending on timestamp.
Steps to Reproduce
- Start docker container
docker run --runtime nvidia -it --privileged --ipc=host --network=host --pid=host -e DISPLAY -v /dev/shm:/dev/shm -v /tmp/:/tmp/ -v /var/nvidia/nvcam/settings/:/var/nvidia/nvcam/settings/ -v ${HOME}/zed_docker_ai/:/usr/local/zed/resources/ CONTAINER_TAG
- Start ZED-X node
roslaunch zed_wrapper zedx.launch &
- Check up timestamps (also TF warnings)
rostopic echo zedx/zed_node/rgb/image_rect_color --noarr
Expected Result
Timestamp changing for all data (depth, RGB, ...)
Actual Result
Same timestamp published all the time
nsec is interchaning between two close values, otherwise stamp doesn't change
There are also warnings from TF on duplicate timestamps
rostopic echo zedx/zed_node/rgb/image_rect_color --noarr
# ...
header:
seq: 192
stamp:
secs: 1698330340
nsecs: 529478000
frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
---
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame base_link at time 1698330340.529479 according to authority unknown_publisher
at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame odom at time 1698330340.529479 according to authority unknown_publisher
at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
header:
seq: 193
stamp:
secs: 1698330340
nsecs: 529479000
frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
---
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame base_link at time 1698330340.529478 according to authority unknown_publisher
at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame odom at time 1698330340.529478 according to authority unknown_publisher
at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
header:
seq: 194
stamp:
secs: 1698330340
nsecs: 529478000
frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
---
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame base_link at time 1698330340.529479 according to authority unknown_publisher
at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame odom at time 1698330340.529479 according to authority unknown_publisher
at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
header:
seq: 195
stamp:
secs: 1698330340
nsecs: 529479000
frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
---
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame base_link at time 1698330340.529478 according to authority unknown_publisher
at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
Warning: TF_REPEATED_DATA ignoring data with redundant timestamp for frame odom at time 1698330340.529478 according to authority unknown_publisher
at line 278 in /tmp/binarydeb/ros-noetic-tf2-0.7.6/src/buffer_core.cpp
header:
seq: 196
stamp:
secs: 1698330340
nsecs: 529478000
frame_id: "zedx_left_camera_optical_frame"
height: 600
width: 960
encoding: "bgra8"
is_bigendian: 0
step: 3840
data: "<array type: uint8, length: 2304000>"
ZED Camera model
ZED-X
Environment
OS
- Host - L4T R35 2.1
- Docker - L4T R35 3
- to be investigated if above mismatch has any impact (but will take hours)
CPU/GPU - Jetson AGX Orin
ZED SDK
- Host - `ZED SDK for JetPack 5.1 (L4T 35.2) 4.0 (Jetson Xavier, Orin AGX/NX 16GB, CUDA 11.4)`
- Docker- image based on [stereolabs/zed:4.0-runtime-l4t-r35.3](https://github.com/stereolabs/zed-docker/blob/fde3fe833859eb78831b0cc4c0a119b745f6ff62/4.X/l4t/runtime/Dockerfile#L20)
- so likely `ZED SDK for JetPack 5.1.1 (L4T 35.3)4.0 (Jetson Xavier, Orin AGX/NX/Nano, CUDA 11.4)`
- mismatch to be investigated (but it will take hours)
ROS Noetic installed from apt
zed_ros_wrapper tested
- current master 71eb2bb434f059e17191503b707267938f5a1b7f (as of 27 Oct 2023)
- but also own fork based on 50dda077a5ea72b7f8c00faad5746938331e46c2 (10th May 2023)
GMSL driver (Docker host)
sudo dpkg -i stereolabs-zedx_0.5.1-MAX96712-L4T35.2_arm64.deb
Anything else?
Docker image is build following way
# https://github.com/stereolabs/zed-docker/tree/master/4.X/l4t
FROM stereolabs/zed:4.0-runtime-l4t-r35.3
# mainly follow with ROS Noetic install from apt
As next step I am going to build from matching L4T + ZED-SDK version (between host and docker) and re-test but this will take hours.
Docker image was cross-built on x86_64
Hi,
It's a mismatch between the L4T version of your host (35.2) and your container (35.3). We'll make more clear in the documentation that a match is required.
Thanks for a quick reply.
Partially expected as I am during image build from matching versions.
Confirmed working when docker host and container L4T match.
From practical point of view it is a bit unfortunate:
- as this means that host and L4T versions have to match
- which defeats some (not all) benefits of using containers
It's a mismatch between the L4T version of your host (35.2) and your container (35.3). We'll make more clear in the documentation that a match is required.
I expect reason to be some hardware/driver/container/SDK interaction.
Nonetheless it would be great to understand reason and extent of potential (in)compatibility
- in the context of long-term software maintenance
- in the context of future coexistence with other software (and dependence on L4T versions)
Hi,
The reason behind is that the driver and user-land SDKs are built with respect to NVIDIA Argus API structure on both side (kernel && user)
Since it has changed between L4T 35.2 and 35.3 (on the timestamp structure especially) which is "only" binary changes, there is no errors triggered but the value gets invalid.
Normally it should not change much starting from 35.3 but that's on NVIDIA hands.
Thank you kindly for above clarification.
It has some impact as this means that:
- matching L4T has guaranteed compatibility
- non-matching L4T has potential compatibility which can be traced and tested