Unable to relaunch the node successful after stopping the node with ctrl + C

Question

Unable to relaunch the node successful after stopping the node with ctrl + C

Closed this issue 6 months ago · 6 comments

Describe the bug
After plugging in the 3DMGX5-AHRS I open a terminal and successfully run the node from the launch file. I then successfully configure and activate the node.

The issue starts when I ctrl + C out of the running node.

Without unplugging the 3DMGX5-AHRS from the usb port I then relaunch the node from the terminal. I then cannot configure the node.

I have to unplug the 3DMGX5-AHRS from the usb port and then plug it back in to get it to successfully configure after exiting with ctrl + C from a successful run. I have not been able to execute two consecutive successful runs.

The code I am using is completely unmodified and taken directly from the ros2 branch.

To Reproduce
Steps to reproduce the behavior:

Unplug and plug back in the 3DMGX5-AHRS
Launch Node in terminal 1:
rosdev@ros2:/ws$ ros2 launch microstrain_inertial_driver microstrain_launch.py
[INFO] [launch]: All log files can be found below /home/rosdev/.ros/log/2023-06-22-22-19-56-285176-ros2-2923
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [microstrain_inertial_driver_node-1]: process started with pid [2935]
Configure the node in terminal 2:
rosdev@ros2:/ws$ ros2 lifecycle set /microstrain_inertial_driver configure
Transitioning successful
Output in terminal 1 from step 3:
[microstrain_inertial_driver_node-1] [INFO] [1687472402.745780235] [microstrain_inertial_driver]: Running microstrain_inertial_driver version: unknown
[microstrain_inertial_driver_node-1] [INFO] [1687472402.745931487] [microstrain_inertial_driver]: Using MIP SDK version: v0.0.0
[microstrain_inertial_driver_node-1] [INFO] [1687472402.746082239] [microstrain_inertial_driver]: Attempting to open serial port </dev/ttyACM0> at <115200>
[microstrain_inertial_driver_node-1] [INFO] [1687472402.747326799] [microstrain_inertial_driver]: Setting device to idle in order to configure
[microstrain_inertial_driver_node-1] [INFO] [1687472402.764998603] [microstrain_inertial_driver]: Main Connection Info:
[microstrain_inertial_driver_node-1] #######################
[microstrain_inertial_driver_node-1] Model Name: 3DMGX5-AHRS
[microstrain_inertial_driver_node-1] Serial Number: 6253.152431
[microstrain_inertial_driver_node-1] Firmware Version: 1.1.76
[microstrain_inertial_driver_node-1] #######################
[microstrain_inertial_driver_node-1] [INFO] [1687472402.765137938] [microstrain_inertial_driver]: Note: Not opening aux port because RTK dongle enable was not set to true.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776686734] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss1/aiding_status
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776779402] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss1/antenna_offset_correction
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776788444] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss1/fix
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776796069] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss1/fix_info
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776803319] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss1/odom
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776809319] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss1/rf_error_detection
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776815611] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss1/sbas_info
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776822111] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss1/time_ref
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776828986] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss2/aiding_status
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776836278] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss2/antenna_offset_correction
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776843903] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss2/fix
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776850236] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss2/fix_info
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776856403] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss2/odom
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776862820] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss2/rf_error_detection
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776868862] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss2/sbas_info
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776874737] [microstrain_inertial_driver]: Note: The device does not support publishing the topic gnss2/time_ref
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776898362] [microstrain_inertial_driver]: Note: The device does not support publishing the topic imu/overrange_status
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776909404] [microstrain_inertial_driver]: Note: The device does not support publishing the topic nav/aiding_summary
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776917321] [microstrain_inertial_driver]: Note: The device does not support publishing the topic nav/dual_antenna_status
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776957863] [microstrain_inertial_driver]: Note: The device does not support publishing the topic rtk/status
[microstrain_inertial_driver_node-1] [INFO] [1687472402.776966530] [microstrain_inertial_driver]: Note: The device does not support publishing the topic rtk/status_v1
[microstrain_inertial_driver_node-1] [INFO] [1687472402.824835238] [microstrain_inertial_driver]: Note: The device does not support the GPIO config command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.824884905] [microstrain_inertial_driver]: Note: The device does not support the PPS source command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.824891072] [microstrain_inertial_driver]: Note: The device does not support the odometer settings command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.824896572] [microstrain_inertial_driver]: Note: The device does not support the factory streaming channels setup command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.824940323] [microstrain_inertial_driver]: Note: The device does not support the SBAS settings command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.824947115] [microstrain_inertial_driver]: Note: The device does not support the nmea message format command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.824981615] [microstrain_inertial_driver]: Note: Device does not support the RTK dongle config command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.825138368] [microstrain_inertial_driver]: Setting Declination Source to 2 0.230000
[microstrain_inertial_driver_node-1] [INFO] [1687472402.830698950] [microstrain_inertial_driver]: Note: Device does not support GNSS antenna offsets
[microstrain_inertial_driver_node-1] [INFO] [1687472402.830805326] [microstrain_inertial_driver]: Note: The device does not support the vehicle dynamics mode command.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.830817327] [microstrain_inertial_driver]: Setting heading source to 1
[microstrain_inertial_driver_node-1] [INFO] [1687472402.836657080] [microstrain_inertial_driver]: Note: Not setting initial heading because heading source is not 0
[microstrain_inertial_driver_node-1] [INFO] [1687472402.836722122] [microstrain_inertial_driver]: Setting autoinitialization to 1
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842700211] [microstrain_inertial_driver]: Note: The device does not support the filter adaptive settings command.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842774712] [microstrain_inertial_driver]: Note: The device does not support the filter aiding command.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842779962] [microstrain_inertial_driver]: Note: The device does not support the relative position configuration command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842793837] [microstrain_inertial_driver]: Note: The device does not support the filter speed lever arm command.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842813046] [microstrain_inertial_driver]: Note: The device does not support the wheeled vehicle constraint command.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842817712] [microstrain_inertial_driver]: Note: The device does not support the vertical gyro constraint command.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842822088] [microstrain_inertial_driver]: Note: The device does not support the GNSS antenna calibration command.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842826213] [microstrain_inertial_driver]: Note: The device does not support the next-gen filter initialization command.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.842837838] [microstrain_inertial_driver]: Setting sensor to vehicle rotation euler to [-0.000000, -0.000000, -0.000000]
[microstrain_inertial_driver_node-1] [INFO] [1687472402.848653132] [microstrain_inertial_driver]: Note: The device does not support the reference point lever arm command
[microstrain_inertial_driver_node-1] [INFO] [1687472402.848739550] [microstrain_inertial_driver]: Note: The settings were not saved as startup settings. Power cycling will remove changes from device
[microstrain_inertial_driver_node-1] [INFO] [1687472402.848745634] [microstrain_inertial_driver]: Resetting the filter after the configuration is complete.
[microstrain_inertial_driver_node-1] [INFO] [1687472402.874798394] [microstrain_inertial_driver]: Setting spin rate to <200.000000> hz
Activate the node in terminal 2:
rosdev@ros2:/ws$ ros2 lifecycle set /microstrain_inertial_driver activate
Transitioning successful
Output in terminal 1 from step 5:
[microstrain_inertial_driver_node-1] [INFO] [1687472712.452925924] [microstrain_inertial_driver]: Resuming the device data streams
Enter ctrl + C in terminal 1:
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[microstrain_inertial_driver_node-1] [INFO] [1687472776.293896831] [rclcpp]: signal_handler(signum=2)
[INFO] [microstrain_inertial_driver_node-1]: process has finished cleanly [pid 3001]
rosdev@ros2:/ws$
Relaunch the node in terminal 1:
rosdev@ros2:/ws$ ros2 launch microstrain_inertial_driver microstrain_launch.py
[INFO] [launch]: All log files can be found below /home/rosdev/.ros/log/2023-06-22-22-27-15-340205-ros2-3046
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [microstrain_inertial_driver_node-1]: process started with pid [3058]
Configure the node in terminal 2:
rosdev@ros2:/ws$ ros2 lifecycle set /microstrain_inertial_driver configure
Transitioning failed
Output from terminal 1 from step 9.:
[microstrain_inertial_driver_node-1] [INFO] [1687472871.684990595] [microstrain_inertial_driver]: Running microstrain_inertial_driver version: unknown
[microstrain_inertial_driver_node-1] [INFO] [1687472871.685062055] [microstrain_inertial_driver]: Using MIP SDK version: v0.0.0
[microstrain_inertial_driver_node-1] [INFO] [1687472871.685124889] [microstrain_inertial_driver]: Attempting to open serial port </dev/ttyACM0> at <115200>
[microstrain_inertial_driver_node-1] [INFO] [1687472871.687295412] [microstrain_inertial_driver]: Setting device to idle in order to configure
[microstrain_inertial_driver_node-1] [ERROR] [1687472877.696646120] [microstrain_inertial_driver]: Unable to set device to idle
[microstrain_inertial_driver_node-1] [ERROR] [1687472877.696747788] [microstrain_inertial_driver]: Error(-4): Timed out
[microstrain_inertial_driver_node-1] [ERROR] [1687472877.696810705] [microstrain_inertial_driver]: Failed to read configuration for node
[microstrain_inertial_driver_node-1] [ERROR] [1687472877.696822539] [microstrain_inertial_driver]: Failed to configure node base

Expected behavior
I expect to see the node successfully configured in step 8 as I do in step 2.

Environment (please complete the following information):
The code is being run inside a Docker container in a Parallels VM running Ubuntu on a Mac M1.

Docker
- OS: Ubuntu 22.04.2 LTS
- Architecture: aarch64
Parallels VM
- OS: Ubuntu 22.04.2 LTS
- Architecture: aarch64
Mac M1 (host)
- OS: macOS 13.4
- Kernel: Darwin 22.5.0
- Architecture: arm64
ROS Version: humble
version: latest
Sensor: 3DM-GX5-AHRS
microstrain_inertial branch: origin/ros2

Modifications
No modifications

Launch Parameters
no updates to the params.yaml

Answer 1 · 2023-07-06T14:04:29.000Z

Could we have an update on this from Microstrain?

Answer 2 · 2023-07-06T15:16:10.000Z

Hello,

Our primary ROS engineer is out currently, but I'll do what I can...

The environment is a bit complex for us to recreate, but is it possible for your system to shutdown the lifecycle node before you Ctrl+C? It's not clear how the USB connection is hanging, but this may be a way of preventing this. We have heard of other customers using VMs have somewhat similar issues with USB connections and it is hard to fully support that type of environment given some of the complexities of how USB is handled in them.

Hope that helps,

Nathan

Answer 3 · 2023-07-11T13:24:03.000Z

Hi Nathan,
Definitely understand there is the VM layer. That could be the issue.
As a data point: we have joysticks and other USB peripherals connected and they all seem ok.

I''d like to keep this open. Maybe we could provide your team with a docker container to reproduce? We see similar issues from others, but they get closed as stale.

For the moment, the below will work, before CTRL+C, as you suggested.
ros2 lifecycle set /microstrain_inertial_driver shutdown

The other option is creating a node that manages the IMU's lifecycle, but we would like to avoid that.

Thank you.

Answer 4 · 2023-07-12T12:34:02.000Z

Sounds good, let us know if you require further assistance!

Nathan

Answer 5 · 2024-04-14T00:59:52.000Z

This issue is stale because it has been open for 2 weeks with no activity. If the issue is still not resolved, please leave a comment describing what is still not working

Answer 6 · 2024-05-05T00:50:23.000Z

This issue was closed because it has been inactive for 2 weeks since being marked as stale. If the issue is still not resolved, please reopen the issue, and leave a comment describing what is still not working