usb_cam silently accepts `pixel_format:=uyvy` / yuv422, even when not supported by the device, and sends images with the wrong `encoding` value
jtbandes opened this issue · 12 comments
(tested on Iron, built from source on latest usb_cam ros2 branch, Ubuntu 22.04 in a VM with macOS arm64 host)
With default settings or pixel_format:=yuyv, the resulting Image messages have encoding: yuv422_yuy2. RViz and Foxglove Studio can display these correctly:
But when I set pixel_format:=uyvy, the Image has encoding: yuv422, and it looks like this:
It seems like maybe what is happening is that the pixel data does not actually change between pixel_format:=yuyv and pixel_format:=uyvy, but only the image message's encoding changes.
@jtbandes so this is a result of the rewrite I did to the pixel format logic and I understand it probably is a bit confusing for anyone not super familiar with the source code. I'm working on improving the documentation related to it.
So one important concept is the separation between the capture format and the published format. For both uyvy and yuv422 there are two different base format classes. One of the classes do not do any conversion - meaning the capture format and the publish format are the same (either uyvy or yuv422). The other class does do a conversion, meaning it captures in either uyvy or yuyv format and publishes in another (rgb).
Most ROS visualization tools do not support visualizing uyvy or yuv422 images directly and instead only support rgb formats (at least from what I've been able to figure out).
So if you'd like to just visualize images in rviz2 or rqt_image_view I'd recommend to use the conversion classes then.
You can do this by specifying in your params file uyvy2rgb or yuyv2rgb. Either should capture in one format, convert the image to rgb and publish that out to the ROS world.
Hope this helps, and let me know if you are still having issues.
oh. actually wait you are saying that for yuyv it is able to work but for uyvy its not. Let me dig in...
@jtbandes ah so it looks like this was just recently added to the supported image encodings. I'll update this repos constants then, maybe it'll fix your issue.
Hm, indeed it seems like rviz2 and rqt_image_view don't actually support the uyvy and yuyv strings yet (I'm using iron; edit: just tested rolling and got the same result as iron), so #265 doesn't help. (They do support yuv422 and yuv422_yuy2 strings which is how I noticed the issue.)
But I think my main point is that setting pixel_format:=yuyv or uyvy doesn't seem to actually have changed the capture format... I think that the same pixel data is being put in the image message for both of these, which means that one of them displays correctly and one does not.
@jtbandes does your device support capturing in both formats? This driver should print to the console what formats your device supports. Can you share it here?
@jtbandes going to close this since no response. Feel free to reopen it if this is still an issue for you.
Sorry for the delay, I was on vacation.
does your device support capturing in both formats? This driver should print to the console what formats your device supports. Can you share it here?
No, I guess it doesn't. This is the full output:
jacob@parallels-jammy:~/Desktop/ros2_ws$ ros2 run usb_cam usb_cam_node_exe --ros-args -p pixel_format:=uyvy -p video_device:=/dev/video2
[INFO] [1692056422.608974546] [usb_cam]: camera_name value: default_cam
[WARN] [1692056422.609487650] [usb_cam]: framerate: 30.000000
[INFO] [1692056422.611439995] [usb_cam]: using default calibration URL
[INFO] [1692056422.612138243] [usb_cam]: camera calibration URL: file:///home/jacob/.ros/camera_info/default_cam.yaml
[ERROR] [1692056422.612228774] [camera_calibration_parsers]: Unable to open camera calibration file [/home/jacob/.ros/camera_info/default_cam.yaml]
[WARN] [1692056422.612362466] [usb_cam]: Camera calibration file /home/jacob/.ros/camera_info/default_cam.yaml not found
[INFO] [1692056422.612402961] [usb_cam]: Starting 'default_cam' (/dev/video2) at 640x480 via mmap (uyvy) at 30 FPS
[INFO] [1692056422.680840598] [usb_cam]: This devices supproted formats:
[INFO] [1692056422.680983414] [usb_cam]: YUYV 4:2:2: 1024 x 576 (15 Hz)
[INFO] [1692056422.680990079] [usb_cam]: YUYV 4:2:2: 1280 x 720 (10 Hz)
[INFO] [1692056422.680992996] [usb_cam]: YUYV 4:2:2: 1600 x 896 (7 Hz)
[INFO] [1692056422.680995829] [usb_cam]: YUYV 4:2:2: 160 x 120 (30 Hz)
[INFO] [1692056422.680998412] [usb_cam]: YUYV 4:2:2: 160 x 90 (30 Hz)
[INFO] [1692056422.681001078] [usb_cam]: YUYV 4:2:2: 176 x 144 (30 Hz)
[INFO] [1692056422.681003494] [usb_cam]: YUYV 4:2:2: 320 x 180 (30 Hz)
[INFO] [1692056422.681006119] [usb_cam]: YUYV 4:2:2: 320 x 240 (30 Hz)
[INFO] [1692056422.681008786] [usb_cam]: YUYV 4:2:2: 352 x 288 (30 Hz)
[INFO] [1692056422.681011244] [usb_cam]: YUYV 4:2:2: 432 x 240 (30 Hz)
[INFO] [1692056422.681014368] [usb_cam]: YUYV 4:2:2: 640 x 480 (30 Hz)
[INFO] [1692056422.681016868] [usb_cam]: YUYV 4:2:2: 800 x 448 (30 Hz)
[INFO] [1692056422.681019409] [usb_cam]: YUYV 4:2:2: 800 x 600 (24 Hz)
[INFO] [1692056422.681021909] [usb_cam]: YUYV 4:2:2: 864 x 480 (24 Hz)
[INFO] [1692056422.681024409] [usb_cam]: YUYV 4:2:2: 960 x 720 (15 Hz)
[INFO] [1692056422.681026992] [usb_cam]: Y/CbCr 4:2:0: 1024 x 576 (30 Hz)
[INFO] [1692056422.681030491] [usb_cam]: Y/CbCr 4:2:0: 1280 x 720 (30 Hz)
[INFO] [1692056422.681032908] [usb_cam]: Y/CbCr 4:2:0: 1600 x 896 (30 Hz)
[INFO] [1692056422.681035324] [usb_cam]: Y/CbCr 4:2:0: 160 x 120 (30 Hz)
[INFO] [1692056422.681037949] [usb_cam]: Y/CbCr 4:2:0: 160 x 90 (30 Hz)
[INFO] [1692056422.681040365] [usb_cam]: Y/CbCr 4:2:0: 176 x 144 (30 Hz)
[INFO] [1692056422.681042823] [usb_cam]: Y/CbCr 4:2:0: 1920 x 1080 (30 Hz)
[INFO] [1692056422.681045573] [usb_cam]: Y/CbCr 4:2:0: 320 x 180 (30 Hz)
[INFO] [1692056422.681047947] [usb_cam]: Y/CbCr 4:2:0: 320 x 240 (30 Hz)
[INFO] [1692056422.681050489] [usb_cam]: Y/CbCr 4:2:0: 352 x 288 (30 Hz)
[INFO] [1692056422.681052863] [usb_cam]: Y/CbCr 4:2:0: 432 x 240 (30 Hz)
[INFO] [1692056422.681055238] [usb_cam]: Y/CbCr 4:2:0: 640 x 360 (30 Hz)
[INFO] [1692056422.681057571] [usb_cam]: Y/CbCr 4:2:0: 640 x 480 (30 Hz)
[INFO] [1692056422.681060446] [usb_cam]: Y/CbCr 4:2:0: 800 x 448 (30 Hz)
[INFO] [1692056422.681062946] [usb_cam]: Y/CbCr 4:2:0: 800 x 600 (30 Hz)
[INFO] [1692056422.681065320] [usb_cam]: Y/CbCr 4:2:0: 864 x 480 (30 Hz)
[INFO] [1692056422.681067945] [usb_cam]: Y/CbCr 4:2:0: 960 x 720 (30 Hz)
[INFO] [1692056422.681071361] [usb_cam]: Setting 'brightness' to 50
unknown control 'brightness'
unknown control 'white_balance_temperature_auto'
[INFO] [1692056422.690110048] [usb_cam]: Setting 'white_balance_temperature_auto' to 1
[INFO] [1692056422.690173541] [usb_cam]: Setting 'exposure_auto' to 3
unknown control 'exposure_auto'
[INFO] [1692056422.696893678] [usb_cam]: Setting 'focus_auto' to 0
unknown control 'focus_auto'
[INFO] [1692056422.704036013] [usb_cam]: Timer triggering every 33 ms
^C[INFO] [1692056439.905122082] [rclcpp]: signal_handler(signum=2)
[WARN] [1692056439.905484797] [usb_cam]: Shutting down
[ros2run]: Segmentation fault
But if a format is not supported by the device, then the usb_cam node probably shouldn't silently pretend like it was (sending the yuyv image data in a message that claims to have encoding: uyvy). It should instead either throw an error and refuse to launch with the invalid configuration, or it should fall back to a supported encoding (but report the correct encoding in the image message's encoding field).
But if a format is not supported by the device, then the usb_cam node probably shouldn't silently pretend like it was (sending the yuyv image data in a message that claims to have encoding: uyvy). It should instead either throw an error and refuse to launch with the invalid configuration, or it should fall back to a supported encoding (but report the correct encoding in the image message's encoding field).
@jtbandes this is good feedback. We should be able to improve the warning here and fall back to another supported format. Let me see if I can get something done quickly to address this 👍🏼
@jtbandes going to close this issue as I believe I've implemented an acceptable solution. Feel free to open another one if the issue still isn't resolved.
👍 the proposed solution sounds fine (I just haven't gotten a chance to test it myself yet). Thanks!