Blackfly S GigE-Frame drops after running with ROS2 : INCOMPLETE 100%
kavishmshah opened this issue · 14 comments
Hi,
I'm working with another FLIR camera and this one is FLIR gigE (Blackfly S GigE BFS-PGE-19S4C-C: 2.0 MP). I connected the camera using a switch over POE port and hooked it onto my PC.
- Spinview recognizes the camera and shows me the feed.
- I followed it up with running the ROS2 node and it gives me: rate [Hz] in 40.00 out 0.00 drop 0% INCOMPLETE 100% as the output.
- I then shut the ROS2 node and ran spinview again and see a black screen with no camera feed.
Below are the parameters as in the launch file:
example_parameters = {
'blackfly_s': {
'debug': False,
'compute_brightness': False,
'adjust_timestamp': True,
'dump_node_map': False,
# set parameters defined in blackfly_s.yaml
'gain_auto': 'Continuous',
'pixel_format': 'BayerRG8',
'exposure_auto': 'Continuous',
# to use a user set, do this:
'user_set_selector': 'UserSet0',
'user_set_load': 'Yes',
# These are useful for GigE cameras
'device_link_throughput_limit': 380000000,
'gev_scps_packet_size': 9000,
# ---- to reduce the sensor width and shift the crop
# 'image_width': 1408,
# 'image_height': 1080,
# 'offset_x': 16,
# 'offset_y': 0,
# 'binning_x': 1,
# 'binning_y': 1,
# 'connect_while_subscribed': True,
'frame_rate_auto': 'On',
'frame_rate': 40.0,
'frame_rate_enable': True,
'buffer_queue_size': 10,
'trigger_mode': 'Off',
'trigger_source': 'Line3',
'line2_selector': 'Line2',
'trigger_activation': 'RisingEdge',
'chunk_mode_active': True,
'chunk_selector_frame_id': 'FrameID',
'chunk_enable_frame_id': True,
'chunk_selector_exposure_time': 'ExposureTime',
'chunk_enable_exposure_time': True,
'chunk_selector_gain': 'Gain',
'chunk_enable_gain': True,
'chunk_selector_timestamp': 'Timestamp',
'chunk_enable_timestamp': True,
},
Below are the screencasts (split nto 2 screen recordings, but they are just followed one after the other). Any way to resolve this?
Screencast from 07-25-2024 09:45:43 AM.webm
Screencast from 07-25-2024 09:46:48 AM.webm
Thanks
Kavish
It doesn't seem to like the device_link_throughput_limit': 380000000 setting. That somehow messes up the GigE link, leading to 100% dropped packets and black image in SpinView afterwards.
Recommendation: reset the camera in SpinView and leave out the device_link_throughput_limit next time you start the ROS driver.
So, I tried leaving out the device_link_throughput_limit and I still see the same issue persist of black image. Do you have any other recommendation?
Below is the terminal output.
[camera_driver_node-1] [INFO] [1721944590.139740086] [Spinnaker Wrapper]: Found camera [serial: 23604040] from: [GEV Interface 2]
[camera_driver_node-1] [INFO] [1721944590.139877538] [23604040]: found camera with serial number: 23604040
[camera_driver_node-1] [INFO] [1721944590.966263865] [23604040]: setting UserSetControl/UserSetSelector to: UserSet0
[camera_driver_node-1] [INFO] [1721944590.968796766] [23604040]: executing UserSetControl/UserSetLoad
[camera_driver_node-1] [INFO] [1721944590.988158297] [23604040]: setting ImageFormatControl/PixelFormat to: BayerRG8
[camera_driver_node-1] [INFO] [1721944591.004467945] [23604040]: setting AnalogControl/GainAuto to: Continuous
[camera_driver_node-1] [INFO] [1721944591.013704346] [23604040]: setting TransportLayerControl/GigEVision/GevSCPSPacketSize to: 9000
[camera_driver_node-1] [INFO] [1721944591.023179378] [23604040]: setting DigitalIOControl/LineSelector to: Line2
[camera_driver_node-1] [INFO] [1721944591.026978963] [23604040]: setting AcquisitionControl/ExposureAuto to: Continuous
[camera_driver_node-1] [INFO] [1721944591.035382733] [23604040]: setting AcquisitionControl/AcquisitionFrameRateEnable to: 1
[camera_driver_node-1] [INFO] [1721944591.040210285] [23604040]: setting AcquisitionControl/AcquisitionFrameRate to: 40
[camera_driver_node-1] [INFO] [1721944591.049087543] [23604040]: setting AcquisitionControl/TriggerMode to: Off
[camera_driver_node-1] [INFO] [1721944591.053845840] [23604040]: setting AcquisitionControl/TriggerSource to: Line3
[camera_driver_node-1] [INFO] [1721944591.059697633] [23604040]: setting AcquisitionControl/TriggerActivation to: RisingEdge
[camera_driver_node-1] [INFO] [1721944591.064996556] [23604040]: setting ChunkDataControl/ChunkModeActive to: 1
[camera_driver_node-1] [INFO] [1721944591.071160551] [23604040]: setting ChunkDataControl/ChunkSelector to: FrameID
[camera_driver_node-1] [INFO] [1721944591.075785346] [23604040]: setting ChunkDataControl/ChunkEnable to: 1
[camera_driver_node-1] [INFO] [1721944591.081129411] [23604040]: setting ChunkDataControl/ChunkSelector to: ExposureTime
[camera_driver_node-1] [INFO] [1721944591.082494373] [23604040]: setting ChunkDataControl/ChunkEnable to: 1
[camera_driver_node-1] [INFO] [1721944591.086894676] [23604040]: setting ChunkDataControl/ChunkSelector to: Gain
[camera_driver_node-1] [INFO] [1721944591.088567981] [23604040]: setting ChunkDataControl/ChunkEnable to: 1
[camera_driver_node-1] [INFO] [1721944591.092726050] [23604040]: setting ChunkDataControl/ChunkSelector to: Timestamp
[camera_driver_node-1] [INFO] [1721944591.094061731] [23604040]: setting ChunkDataControl/ChunkEnable to: 1
[camera_driver_node-1] [INFO] [1721944591.287892157] [23604040]: camera has pixel format: BayerRG8
[camera_driver_node-1] [INFO] [1721944591.287983221] [flir_camera]: camera_driver_node started up!
[camera_driver_node-1] [WARN] [1721944591.439719981] [23604040]: rate [Hz] in 30.05 out 0.00 drop 0% INCOMPLETE 100%
[camera_driver_node-1] [WARN] [1721944596.439879324] [23604040]: rate [Hz] in 35.51 out 0.00 drop 0% INCOMPLETE 100%
[camera_driver_node-1] [WARN] [1721944601.439686473] [23604040]: rate [Hz] in 39.33 out 0.00 drop 0% INCOMPLETE 100%
[camera_driver_node-1] [WARN] [1721944606.439724277] [23604040]: rate [Hz] in 39.90 out 0.00 drop 0% INCOMPLETE 100%
Did you reset the camera? There is some option in SpinView to do that. No point trying the ROS driver until you have it working in SpinView again.
I have it working in spinview again.
The above was after resetting the camera through spinview.
Another thing: I see you are loading userset 0. That one may not be what you want. Can you take that out (or check that loading userset 0 works in spinview and does not mess up your camera)?
I tried running the ROS driver without setting the throughtput limit after the camera was streaming perfectly on Spinview and the black screen shows up.
As for the userset, I checked it in spinview and userset 0 works fine. Also, commented out userset 0 in the launch file of the ROS driver, the same black screen shows up.
I have only seen the "incomplete" warnings on GigE when somehow the ethernet link was misconfigured. I really don't know what is happening in your situation.
I was thinking it might be something with the switch I'm using, but then, also see Spinview working fine. Not sure what is causing this issue.
I also tried the same setup with an NVIDIA devkit and see the INCOMPLETE 100% output.
You can try to revert to an earlier tag where I had not implemented the incomplete checking, maybe there's something broken with the check. But if it's incomplete then, it will just not publish, without any indication what's broken.
I get 100% incomplete when I set the 'gev_scps_packet_size':` 9000, but leave the network card is still at MTU 1500.
Can you check your network card settings (ip a should tell).
Got it working at MTU 9000 by setting the MTU on the host, and using a newer switch. Some users have reported you have to set the MTU to a bit larger than 9000 on the switch. My switch seems automatically do the right thing.
BTW my setup works fine with the network cards at MTU 1500, so long as I don't set gev_scps_packet_size to 9000 in the ROS config. Maybe your SpinView did not ask for Jumbo packets, that's why it works with your cards/switch at 1500.
I assume this problem is due to a misconfigured MTU on the network card. Closing. Please reopen if still a problem.
@berndpfrommer Apologies for the delayed response. The above solution of configuring MTU did not solve our issue. But we were able to solve the bandwidth issue using multicast.
I will share the steps soon as we replicate it on multiple systems.
Thanks!