Azimuth range missing from lidar scan / images
Qgel opened this issue · 5 comments
I'm having an issue where about 30-50 degree azimuth of the lidar scan are missing from the frames published by the driver. This happens for both the point clouds, as well as the published images.
I'm using an Ouster OS1-128 FW 2.2.1 on ros rolling on Ubuntu 20.04.
I have created a short video of the ambient image which shows the issue:
https://user-images.githubusercontent.com/1146690/159955795-6f74cdc6-dc9c-4517-9ce7-4a72632ad7de.mp4
From this, it looks like the start of the missing area is always in the same place, and the end of the missing area varies by maybe 20 degrees azimuth. The ambient image also looks wrong in general.
I have tested multiple different OS1-128 sensors, and they all show this behavior. When I look at the output in OusterStudio on the same system, the issue is not present.
If I revert commit 239cd09 the behavior improves, but the issue is still somewhat present:
https://user-images.githubusercontent.com/1146690/159959865-c4f86d4c-cc25-471c-aae5-9db1bffe2fed.mp4
There is still periodically missing data, but there is no 'constant' missing data. Also the ambient image looks correct now.
I have the same problem with the OS1-64 (Rev D) FW 2.2 and 2.2.1 (haven't tested any other version but for the past 5 months this has happened to us as well). I get more of the second video where it sometimes works perfectly and sometimes it flickers in a particular section.
I have managed to reduce how often it happens by reducing the Azimuth window. Perhaps it has something to do with the amount of data. Maybe it is dropping messages or something like that.
For me it happens for all messages, Image and pointcloud so it really seems like it is just dropping messages constantly. I tried multiple cables and even the fastest ethernet cards and although it seems to improve it never completely fixes the problem
I looked into this issue in some more detail and actually just finished a pull request a few minutes after you commented ;)
First of all, I was wrong that commit #239cd09 has anything to do with this issue. I think I was simply switching between 2048x10 and 1024x10 mode between tests, which made the issue slightly less pronounced for me.
I think the core of the issue is that currently, the processing function both receives the data and then runs the lidar processing once a scan is complete. On my system, this was blocking the processing function for about 20 ms, during which time new packets would be dropped. At 10 Hz this is 20% of our time, and pretty nicely fits with the observation of about ~15% of lidar data missing. During my tests the problem also seemed load-dependent, which also makes sense.
I just created a pull request #97 which splits receiving data from the wire and processing data into two separate threads through a ringbuffer. This completely fixes the issue on my system:
ouster_buffered.mp4
Great news! I was adding the Azimuth window as a parameter since the most recent version of the firmware seems to now be able to save it after a reboot. I might still do a PR on it but I prefer your solution for sure
#98 His PR for reference. I don't have an issue with the implementation, but it touches code that must be brought back to the core ouster client code First so that we stay in sync with that here so that we get out of the box similar support for ROS 1 and ROS 2.