LORD-MicroStrain/microstrain_inertial

Invalid NMEA sentences and RTCM write error

Opened this issue · 2 comments

taskbjorn commented

Describe the bug
On my setup, NMEA sentences published by the GQ7 are occasionally empty or do not pass the checksum. This usually happens after some time (ranging from minutes to hours, but happens more often with high NMEA data rates), the NTRIP client log shows either of these two blocks of warnings, relating to:

  • A missing checksum
[ntrip_ros.py-9] [WARN] [1716296096.106033426] [ntrip_client]: Received invalid NMEA sentence. Sentence should have a "*" character to seperate the checksum, but we could not find it.
[ntrip_ros.py-9] [WARN] [1716296096.107601044] [ntrip_client]: Sentence: !�6h
[ntrip_ros.py-9] [WARN] [1716296096.108954746] [ntrip_client]: Invalid NMEA sentence, not sending to server
  • An invalid checksum
[ntrip_ros.py-9] [WARN] [1716296096.108954746] [ntrip_client]: Invalid NMEA sentence, not sending to server
[ntrip_ros.py-9] [WARN] [1716296867.293565107] [ntrip_client]: Found packet, but checksums didn't match
[ntrip_ros.py-9] [WARN] [1716296867.294513096] [ntrip_client]: Expected Checksum: 0x25B607
[ntrip_ros.py-9] [WARN] [1716296867.295463915] [ntrip_client]: Actual Checksum: 0x28B382

Some time after the first warnings appear, I can read a truncated error message from the microstrain_inertial_driver_node regarding a SIGHUP signal being received by the MIP SDK, and immediately after the following error message is flooded to the console:

[microstrain_inertial_driver_node-8] [ERROR] [1716317808.192041445] [gnss_ins]: Failed to write RTCM to device

From there onwards, the node is unable to process RTCM data and does not recover.

To Reproduce
Steps to reproduce the behavior:

  1. Run the microstrain_inertial_driver_node lifecycle node from the microstrain_inertial_driver package.
  2. Run the ntrip_ros.py node from the ntrip_client package.
  3. Wait for some time (a few minutes to a few hours).
  4. Observe the ntrip_ros log for NMEA warnings.
  5. Observe the microstrain_inertial_driver_node for RTCM write errors.

Expected behavior
I did not expect to read empty or corrupted NMEA sentences from the GQ7 AUX port. I also did not expect the ROS node to stop receiving RTCM data altogether with no attempt to recover.

Environment:

  • OS: Ubuntu 22.04
  • Architecture: x86_64
  • ROS Version: Humble Hawksbill
  • Version: 4.2.0
  • Sensor: 3DM-GQ7

Modifications
No modifications to source code, using the latest package ros-humble-microstrain-inertial-driver from the ROS Humble Hawksbill distribution.

Launch Parameters
See attached params.yml file.

Additional context
I am connecting to the GQ7 via two USB to DB9 cables, going from two USB 2.0 ports of my Linux machine running both ROS nodes to the MAIN and AUX ports of my GQ7. When ntrip_interface_enableand nmea_message_config are disabled , the sensor works perfectly.

I have tried the following:

  1. Switching to different ports did not make a difference.
  2. Changing baudrates on the MAIN and AUX ports did not make a difference.
  3. Removing all published NMEA sentences but one (gnss1_nmea_gga_data_rate) reduced the rate at which warnings appear, but did not eliminate the issue.
robbiefish commented

Hi @taskbjorn,

I am taking a look at this now, but am currently unable to reproduce this.

I will note that these logs

[ntrip_ros.py-9] [WARN] [1716296096.108954746] [ntrip_client]: Invalid NMEA sentence, not sending to server
[ntrip_ros.py-9] [WARN] [1716296867.293565107] [ntrip_client]: Found packet, but checksums didn't match
[ntrip_ros.py-9] [WARN] [1716296867.294513096] [ntrip_client]: Expected Checksum: 0x25B607
[ntrip_ros.py-9] [WARN] [1716296867.295463915] [ntrip_client]: Actual Checksum: 0x28B382

Are not very well formed, but are saying that we received an Invalid NMEA sentence, and then, we received an RTCM message from the NTRIP caster that has an invalid checksum.

This makes me suspicious that this problem could be with the NTRIP caster. Can you provide any information about the caster you are using? Obviously not the username and password, but other information would be helpful. If you are uncomfortable sharing that information on Github, please feel free to reach out to me via email

taskbjorn commented

Thank you for looking into this @robbiefish - I have sent you an email with more details on my NTRIP caster, but feel free to come back here if troubleshooting moves away from the NTRIP caster itself and if the conversation can be helpful to other users.