Getting UxPlay working with hardware decoding on Raspberry Pi.
Closed this issue · 4 comments
EDIT: Problem is now solved by backports from GStreamer-1.21.0.1
Recipes for patching The GStreamer Video4Linux2 plugin from gstreamer1.0-plugins-good are given here for (1) GStreamer-1.18.4 as used on Debian Bullseye-based systems such as Rapberry Pi OS (2) GStreamer-1.18.5 systems such as Ubuntu 21.20, and (3) GStreamer-1.20.0 systems such as Manjaro-Arm-RPi4 (These are for use until your distribution provides these updates).
The instructions have been moved to the UxPlay Wiki.
The problem is explained here:
See also this post
I'm trying to modify a working h264 video pipeline
... ! h264parse ! avdec_h264 ! videoconvert ! (xvimagesink) ...
to ...! h264parse ! v4l2h264dec ! v4l2convert ! (xvimagesink) ... so as to use GPU h264 decoding on Raspberry Pi 4 (64bit, gstreamer 1.20.0 running Linux Manjaro (Arch))
The h264 stream is decrypted from Apple's Airplay and the first frame sent by appsrc is the PPS and SPS which provides caps.
Caps negotiation fails (perhaps because h264parse is asking for colorimetry {1:3:5:1} and v4l2h264dec doesnt offer it, or maybe because profile=high-10 not high (??? wild guesses, looking at GST_DEBUG=GST_CAPS:5 output) and comparing with what happens with avdec-h264 software CPU decoding)
I'm guessing peer query returned EMPTY (below) is the failure(?).
Since video4linux2 is now the only way to access the GPU on the Pi, now that omx is gone, it seems pretty important to have working gstreamer plugins for its broadcom GPU.
Could there be some workaround for this caps problem using capssetter?
probably relevant snip of GST_DEBUG output is here:
0:00:13.327692263 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3155:gst_pad_peer_query_caps:<v4l2h264dec0:src> peer query returned video/x-raw, format=(string)NV12, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 21
47483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)I420, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], i
nterlace-mode=(string)progressive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)YV12, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(
string)progressive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)YUY2, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progress
ive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)UYVY, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetr
y=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)NV12, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ bt7
09, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)I420, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ bt709, bt601, smpt
e240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)YV12, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2,
2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)BGRx, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ 1:1:5:1, 1:1:16:4, 1:1:6:5, 1:1:5:2, 1:1:5:3, sR
GB, 1:1:12:8, 2:1:11:7, 1:1:0:0 }; video/x-raw, format=(string)BGR, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ 1:1:5:1, 1:1:16:4, 1:1:6:5, 1:1:5:2, 1:1:5:3, sRGB, 1:1:12:8, 2:1:11:7,
1:1:0:0 }; video/x-raw, format=(string)RGB, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ 1:1:5:1, 1:1:16:4, 1:1:6:5, 1:1:5:2, 1:1:5:3, sRGB, 1:1:12:8, 2:1:11:7, 1:1:0:0 }; video/x-raw
, format=(string)NV21, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(strin
g)RGB16, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ 1:1:5:1, 1:1:16:4, 1:1:6:5, 1:1:5:2, 1:1:5:3, sRGB, 1:1:12:8, 2:1:11:7, 1:1:0:0 }; video/x-raw, format=(string)GRAY8, width=(int)[
32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)YVYU, width=(int)[ 32, 1920, 2 ],
height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }; video/x-raw, format=(string)GRAY16_LE, width=(int)[ 32, 1920, 2 ], height=(i
nt)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)progressive, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }
0:00:13.328324218 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3155:gst_pad_peer_query_caps:<h264parse0:src> peer query returned video/x-h264, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], colorime
try=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true
0:00:13.328504123 3992 0xaaaae0812e40 DEBUG GST_CAPS gstpad.c:2737:gst_pad_get_current_caps:<h264parse0:src> get current pad caps (NULL)
0:00:13.328723621 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3185:gst_pad_query_accept_caps:<v4l2h264dec0:sink> accept caps of video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uin
t)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2
0:00:13.328856693 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3102:gst_pad_query_caps:<v4l2h264dec0:sink> get pad caps with filter video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(
uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2
0:00:13.328981654 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3148:gst_pad_peer_query_caps:<v4l2h264dec0:src> get pad peer caps with filter video/x-raw, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, colorimetry=(string)1:3:5:1
0:00:13.329160986 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3148:gst_pad_peer_query_caps:<v4l2convert0:src> get pad peer caps with filter EMPTY
0:00:13.329262244 3992 0xaaaae0812e40 DEBUG GST_CAPS gstpad.c:3259:gst_pad_query_caps_default:<video_sink:sink> query caps caps query: 0xffffa4003e30, GstQueryCaps, filter=(GstCaps)EMPTY, caps=(GstCaps)"NULL";
0:00:13.329357613 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3155:gst_pad_peer_query_caps:<v4l2convert0:src> peer query returned EMPTY
0:00:13.329442204 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3155:gst_pad_peer_query_caps:<v4l2h264dec0:src> peer query returned EMPTY
0:00:13.329522018 3992 0xaaaae0812e40 DEBUG GST_CAPS gstutils.c:3109:gst_pad_query_caps:<v4l2h264dec0:sink> query returned EMPTY
0:00:13.329691923 3992 0xaaaae0812e40 WARN GST_CAPS gstpad.c:5757:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-
chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
The issue seems to be that colorimetry=(string)1:3:5:1 listed as a requirement by the video source is not included in the values listed as supported by v4l2h264dec, which are colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }
Can this be worked around with a capssetter item in the pipeline?
some progress:
The GStreamer "Caps negotiation" issue seems to be fixed by
-vd " capssetter replace=true caps=\"video/x-h264, stream-format=(string)byte-stream, alignment=(string)au \" ! v4l2h264dec "
This gets us to another error:
0:00:12.079774506 27133 0xaaaae7b17000 DEBUG GST_CAPS gstpad.c:2709:gst_pad_has_current_caps:<h264parse0:src> check current pad caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high, level=(string)4.2
GStreamer error: No valid frames decoded before end of stream
This error was already reported by icecube25 in raspberrypi/firmware#1673 and is apparently a RPi Broadcom firmware issue, so It may be fixed faster than the timescale for GStreamer fixes to appear. EDIT: NO IT IS NOT A FIRMWARE ISSUE. NOW FIXED IN GStreamer-1.21.0.0
The (partial) fix is
uxplay -vp "h264parse ! capssetter caps=\"video/x-h264, colorimetry=bt709\" " -vd v4l2h264dec -vc v4l2convert
this is now available as an undocumented option -rpi
The rest of the fix will require changes to gstreamer1.0-plugins-good that are now merged in gstreamer's
development branch, and will hopefully be backported to gstreamer-1.18 for Raspberry Pi OS and gstreamer-1.20 manjaro-rpi
Will test to see if these proposed fixes actually solve the issue.
See the UxPlay Wiki for patching instructions