encoding ros image topic and saving

[KibaekJeong]

Hi Brett,

I really appreciate your work!

I am trying to convert ROS2 image topic into a video with h265 encoding here, but keep getting error.

Here is my GStreamer description

descr: 'rosimagesrc ros-topic="/camera/image" ! videoconvert ! video/x-raw, width=1920,height=1080! x265enc ! video/x-h265,stream-format=byte-stream ! h265parse ! filesink location=test.mp4'

when I use V4L2src on my desktop, the script above works fine.

I am using the ros2 USB camera node and publishing the image as ros2 topic

and I get error of :
[pipeline_node-1] [ERROR] [gst_pipeline_node]: 18446744073709551615 error message from rosimagesrc0 (gerror=GLib.Error('Internal data stream error.', 'gst-stream-error-quark', 1), debug='gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:rospipe/GstBin:ros-video/Rosimagesrc:rosimagesrc0:\nstreaming stopped, reason not-negotiated (-4)')

Any idea where this could come from??

I have been trying to debug myself, but having a hard time fixing this issue by myself.

Thank you

[BrettRD]

The error you're seeing is gstreamer saying it can't get two elements to agree on a video format, this is almost certainly between rosimagesrc and x265enc, which you've sensibly tried to mitigate with videoconvert.

I don't think videoconvert can perform image scaling, so it's probably a size mismatch between dimensions of the ros message and the size you've demanded (video/x-raw, width=1920,height=1080). I think it should settle down if you add a videoscale into the pipeline.

If that doesn't solve it, it might be an encoding name translation issue between rosimagesrc and videoconvert, what's the message metadata from ros2 topic?

https://gstreamer.freedesktop.org/documentation/videoconvert/index.html
https://gstreamer.freedesktop.org/documentation/videoscale/index.html

[KibaekJeong]

Thank you for your help! I have just tried adding videoscale into the pipeline, and adjusted video size to match with ros2 image topic metadata.

here is metadata:

header:
stamp:
sec: 1614919145
nanosec: 53672124
frame_id: camera
height: 480
width: 640
distortion_model: plumb_bob
d: [0.260086, -0.025048, 0.089063, 0.138628, 0.0]
k: [944.012173, 0.0, 534.248944, 0.0, 893.42892, 358.594765, 0.0, 0.0, 1.0]
r: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
p: [852.395142, 0.0, 565.89763, 0.0, 0.0, 922.066223, 386.58625, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi:
x_offset: 0
y_offset: 0
height: 0
width: 0
do_rectify: false

After adding videoscale, I get this error:
[pipeline_node-1] [ERROR] [gst_pipeline_node]: 18446744073709551615 error message from x265enc0 (gerror=GLib.Error('Can not initialize x265 encoder.', 'gst-stream-error-quark', 8), debug='gstx265enc.c(699): gst_x265_enc_init_encoder (): /GstPipeline:rospipe/GstBin:ros-video/GstX265Enc:x265enc0')

Now, my pipeline is
rosimagesrc ros-topic="/camera/image" ! videoconvert ! videoscale ! video/x-raw, width=640,height=480! x265enc ! video/x-h265,stream-format=byte-stream ! h265parse ! filesink location=test.mp4

also tried adding format=I420

Thank you for your help!

[KibaekJeong]

When I use x264enc instead of 265, it records very first image, but nothing else. so the file shows very first image and then nothing else is added.

[BrettRD]

A new error! Marvellous!
This might be a framerate issue, ros2 messages have no idea what frame rate the topics operate at, and this might mess with the encoder. videorate will drop or duplicate frames to make sure the encoder stays fed.
https://gstreamer.freedesktop.org/documentation/videorate/index.html

Unrelated to the bug, you might need to run multiple threads on this pipeline to make sure encoding can continue while ros waits for new frames, you can do that with queue

rosimagesrc ros-topic="/camera/image" ! queue ! videoconvert ! videoscale ! videorate ! video/x-raw, width=640, height=480, framerate=15 ! x265enc ! video/x-h265,stream-format=byte-stream ! h265parse ! filesink location=test.mp4

I don't have a setup to test this pipeline

[KibaekJeong]

Thank you for your help, Brett!!
I was able to accomplish what I needed.

FYI, rosimagesrc does support h264 encoding without any error.
I don't know why, but while v4l2src supports h265 without any error in Linux (desktop), rosimagesrc was not able to support h265. I do know that h265 is generally not supported in the system without an encoder. But I think v4l2src somehow handles the situation without an encoder, using x265enc.

On the other hand, in nvidia jetson, I was able to encode in h265 with rosimagesrc, with omxh265enc used in the pipeline. So I was able to achieve what I needed.

( I will be working on your repo as I need rtsp instead of webrtc )

[Aki1608]

Hi @KibaekJeong, were you able to get the rtsp stream? I am also having similar problem as you.