NVIDIA-ISAAC-ROS/isaac_ros_object_detection

DetectNet freezes and stops publishing

Closed this issue · 7 comments

GEngels commented

We have been very happy with your Isaac repo, but yesterday I ran into a small problem. After running the object detector for a couple of minutes the publishing freezes and nothing is published to the topic. If you then a ros command like (ros2 topic list, topic echo, etc) it continues again for a couple of minutes and freezes at some point. By calling a ros command you seem to trigger something and make it restart.

Nothing is publishes to the encoder so there is also nothing at the topic /detectnet/detections, but it seems to already stagnate at /tensor_pub. We are testing with a realsense camera and the image topic of the realsense keeps working, so that doesn't seem to be the problem.

Any idea what could be going wrong?

Ilia-Kovalev commented

I'm a colleague of @GEngels and after some extra investigation I can add the following:

  • the problem disappears after changing component container type to component_container instead of component_container_mt
  • the problem still appears when I launched only two nodes: realsense2_camera and isaac_ros_dnn_encoders
swapnesh-wani-nvidia commented

Thank you for the feedback. Could you please provide more details on how to reproduce the issue? Which platform are you running this on? Which RealSense camera?

GEngels commented

We are using a Jetson AGX Orin with the D455 realsense camera. We are running everything inside a docker container with DP 1.1 Release.

Ilia-Kovalev commented

With DP 2 release the problem occurs too.

jaiveersinghNV commented

Thanks for the additional information, we are looking into this.

GEngels commented

Any update on this?

hemalshahNV commented

We have not been able to reproduce this. Can you see this on other AGX Orin units?

Notes from SQA below.

  • Xavier AGX + isaac_ros_detectnet (public DP2) + realsense camera (D435) [>4Hrs]
  • Xavier AGX + isaac_ros_detectnet (public DP2) + realsense Rosbags (provided rosbag in quickstart, freshly recorded rosbag) [>4Hrs]
  • Orin AGX + isaac_ros_detectnet (public DP2, internal DP_1.1) + realsense Rosbags (provided rosbag in quickstart, freshly recorded rosbag) [>4Hrs]
    Also checked with running just realsense2_camera node and isaac_ros_dnn_encoders ... to see if it freezes (mentioned in #2).. Still no repro.