[3.4.0][libdile_vt] Image crop on all HDMI sources and WebOS streaming app
PaoloVallesi opened this issue · 10 comments
Edit: more info in this comment
Everything works fine while watching tv channels.
On HDMI source or any WebOS app like netflix, disney+, prime video or youtube, video received on hyperion server is cropped to the top left corner, see images below (ignore the text overlay that shows up on chromecast when pausing a video):
On the HDMI source HDR is disabled. but the same issue occurs on netflix or any other streaming app that support HDR.
My configuration (piccap):
{
"returnValue": true,
"backend": "libdile_vt",
"port": 19400,
"width": 192,
"height": 108,
"captureUI": true,
"ip": "192.168.1.201",
"fps": 0,
"backmsg": "Settings got successfully!",
"captureVideo": true,
"autostart": true
}
Hardware:
"board_type": "M16_DVB",
"hardware_id": "HE_DTV_W16N_AFADABAA",
"product_id": "43UH750V-ZA",
"core_os_release": "3.4.0-5908",
"webos_manufacturing_version": "05.60.25"
Also, trying to start hyperion webos directly from a WebOS app or an HDMI source
return this error:
./hyperion-webos -b libdile_vt -a 192.168.1.201 -S --verbose
0.006s [INFO main ] Starting up...
0.006s [ DBG parse_options ] Finished parsing arguments
0.006s [ DBG capture_main ] Using these values: Address: 192.168.1.201 | Port: 19400 | Width: 360 | Height: 180 | FPS: 0 | Backend: libdile_vt | NoVideo: 0 | NoGUI: 0 | Autostart: 0
0.006s [ DBG capture_main ] Detecting backend...
0.006s [ DBG import_backend_library] Full library path: /media/developer/apps/usr/palm/services/org.webosbrew.piccap.service/libdile_vt_backend.so
0.018s [ DBG capture_main ] Backend preinit...
0.018s [INFO capture_preinit ] Preinit called. Copying config..
0.018s [ DBG capture_main ] Initiating capture...
0.018s [INFO capture_init ] Init called
0.019s [ DBG capture_main ] Starting capture..
0.019s [INFO capture_start ] Capture start called.
KADP_DE_SetDHDRInit : Done.
0.082s [ DBG capture_start ] Got DILE_VT context!
0.083s [ DBG capture_start ] supportScaleUp: 1; (1920x1080)
0.083s [ DBG capture_start ] supportScaleDown: 1; (320x240)
0.083s [ DBG capture_start ] maxResolution: 1920x1080
0.083s [ DBG capture_start ] input deinterlace: 0; display deinterlace: 1
0.083s [WARN capture_start ] DISPLAY dump location failed, attempting SCALER...
0.083s [ ERR capture_main ] Error! capture_start. Code: -2
0.083s [ DBG cleanup ] Starting cleanup... isRunning: 0
0.083s [ DBG cleanup ] Destroying hyperion-client...
0.083s [ DBG cleanup ] Cleanup capture within library...
0.085s [ DBG cleanup ] Cleanup finished.
Don't know if it's related...
i tried few moments ago using 320x240, still getting cropped images on HDMI and webos app
I did a few test trying to narrow the field and things got weird.
I notice that the bug seems to occur depending on the image that's been displayed. Eg:
- Movie (with black bars) starts, full picture is shown on hyperion
- After a few scenes change, picture get's cropped, only 2/3 of the original picture is shown
- Again, after a few scenes change, picture get's cropped even more until we get the situation shown on the first post.
i don't know if there is some auto detection of the image that need to be captured... am i way off here?
edit: btw, i'm using this build: https://github.com/webosbrew/hyperion-webos/actions/runs/1741885843
which solved my issue #38
Did a few more test, could this be related to HDR content?
i know i said i've disabled HDR on HDMI but i'm trying some disney+ non HDR content and it looks like the issue is not present.
reading this log [DILE_VT] pixelFormat: 4; width: 192; height: 108; stride: 1920 i can tell the code ends up here:
...
} else if (vfbprop.pixelFormat == DILE_VT_VIDEO_FRAME_BUFFER_PIXEL_FORMAT_YUV422_SEMI_PLANAR) {
if (outbuf == NULL)
outbuf = malloc (3 * width * height);
if (argbvideo == NULL)
argbvideo = malloc(4 * width * height);
if (uplane == NULL)
uplane = malloc(width / 2 * height);
if (vplane == NULL)
vplane = malloc(width / 2 * height);
// This pixel format is called "NV16". While it is not supported as an
// ARGB target in libyuv, we just need to split interlaced uv values
// into two separate planes and this becomes I422 which is well
// supported.
// TODO: we use ARGB intermediate target, since I422ToRGB24 is not a
// thing. This is not much of a problem, since will be used later when
// UI layer blending will be introduced.
SplitUVPlane(vfbs[idx][1], vfbprop.stride, vplane, width/2, uplane, width/2, width/2, height);
I422ToARGB(vfbs[idx][0], vfbprop.stride, uplane, width/2, vplane, width/2, argbvideo, 4 * width, width, height);
ARGBToRGB24(argbvideo, 4 * width, outbuf, 3 * width, width, height);
} else {
...
i'm wondering if the conversion would still be valid for HDR content...
Edit: btw i installed the NDK so i can build. If there is some "on the fly" suggested modification i should try, go ahead.
i've hid the previous comments 'cause i think they are not the right path to resolving this issue.
I'm pretty much convinced that the issue is the 4K resolution.
The pixel format my tv is using is the YUV422 Semiplanar. i did this test:
- Open a 4k video in the youtube app.
- Select the resolution to 720p, then scale up until the issue shows up:
- 720p OK
- 1080p OK
- 1440p minor crop
- 2160p image cropped, only 1/4 of the actual image is trasmitted (from top left corner)
I thought the issue was based on the image shown but that was a wrong assumption. Streaming app start downloading a "safe" resolution for the connectivity (1080p for me) then scale up. When finally the 4K "buffer download" is available the switch is made and for me means, the image get's cropped to 1/4 due to this issue.
i don't now how to confirm my theory with some debug but if someone tell me what changes can i do to prove this i've the NDK installed and i can build, hence test it out.
Just to clarify - when doing these tests - what capture resolution are you using? 1920x1080? Or something similar? Wonder what is the stride value when your image is OK or cropped in that set up.
Also I suspect this whole issue may be caused 0.083s [WARN capture_start ] DISPLAY dump location failed, attempting SCALER... (ie. SCALER capture behaves differently to DISPLAY we use by default) but there's really nothing we can do about this, since DISPLAY is just failing on your platform)
I'm always using a 16:9 aspect ratio, i didn't know if using 4:3 was introducing more complexity to the whole process, to me make sense to use the same aspect ratio of the screen. I can even go lower than the suggested min values which is 320x240.
For testing purposes i used 400x270. Usually it runs with 192x108 with no issue (except for this one ofc).
stride value is set in capture_start() and it's always 1920, but i want to point out that on my platform hyperion_webos only start when "watching" tv channels (max resolution is 1080), if i try to start it while on hdmi or while using a webos app it fails to do so (2nd post, hidden)
Again, dile_vt fail to start while on 4K res.
will the stride update if i put a DILE_VT_GetAllVideoFrameBufferProperty inside capture_frame or it's just pointless to do this? sorry i really don't know anything about image capturing/processing...
edit: i keep writing "stripe" instead of stride...
edit2: tried putting DILE_VT_GetAllVideoFrameBufferProperty inside capture_frame.
Added the following code for debug inside the if that post frame processing/send time stats:
uint32_t** ptr_cf = calloc(sizeof(uint32_t*), vfbcap.numVfbs);
for (int vfb = 0; vfb < vfbcap.numVfbs; vfb++) {
ptr_cf[vfb] = calloc(sizeof(uint32_t), vfbcap.numPlanes);
}
vfbprop_cf.ptr = ptr_cf;
DILE_VT_GetAllVideoFrameBufferProperty(vth, &vfbcap, &vfbprop_cf);
INFO("[DILE_VT] pixelFormat: %d; width: %d; height: %d; stride: %d...", vfbprop_cf.pixelFormat, vfbprop_cf.width, vfbprop_cf.height, vfbprop_cf.stride);
for (int vfb = 0; vfb < vfbcap.numVfbs; vfb++) {
free(ptr_cf[vfb]);
}
free(ptr_cf);
stride is always 1920, i'll paste part of the log:
2.375s [INFO capture_frame ] [DILE_VT] pixelFormat: 4; width: 480; height: 270; stride: 1920...
5.070s [ DBG capture_frame ] [DILE_VT] frame processing time: 6.471ms; frame send time: 146.243ms
5.070s [INFO capture_frame ] [DILE_VT] pixelFormat: 4; width: 480; height: 270; stride: 1920...
5.077s [ DBG capture_frame ] framerate: 6.383166 FPS
7.718s [ DBG capture_frame ] [DILE_VT] frame processing time: 6.358ms; frame send time: 194.574ms
7.718s [INFO capture_frame ] [DILE_VT] pixelFormat: 4; width: 480; height: 270; stride: 1920...
switched to 4K
97.912s [ DBG capture_frame ] [DILE_VT] frame processing time: 8.494ms; frame send time: 140.091ms
97.913s [INFO capture_frame ] [DILE_VT] pixelFormat: 4; width: 480; height: 270; stride: 1920...
100.745s [ DBG capture_frame ] [DILE_VT] frame processing time: 8.711ms; frame send time: 169.644ms
100.746s [INFO capture_frame ] [DILE_VT] pixelFormat: 4; width: 480; height: 270; stride: 1920...
100.767s [ DBG capture_frame ] framerate: 5.483519 FPS
103.523s [ DBG capture_frame ] [DILE_VT] frame processing time: 10.216ms; frame send time: 176.335ms
103.523s [INFO capture_frame ] [DILE_VT] pixelFormat: 4; width: 480; height: 270; stride: 1920...
SOLVED!
@Informatic your hypotesis of the output mode being the issue here was correct. So i went with a brute force approach 😊
in dile_vt.h from the tv-native-apis repo, the documented DUMP LOCATION TYPE are only two:
typedef enum DILE_VT_DUMP_LOCATION_TYPE_T
{
DILE_VT_SCALER_OUTPUT = 0,
DILE_VT_DISPLAY_OUTPUT = 1,
} DILE_VT_DUMP_LOCATION_TYPE_T;
when actually looks like there are more...
so i tried using different value for dump_location variable:
For me, using the value 2 fixes the issue.
- Using
2, the capture start also while on 4K (it didn't with DILE_VT_SCALER_OUTPUT).
Switching from 1080p/i to 2160p resolution doesn't break the capture, the image is always captured completly.
other values i tried:
- 3
- 4
then i got tired since they both gave the same working result as mode 2.
here's my simple change in the code in libdile_vt.c from row 125:
if (DILE_VT_SetVideoFrameOutputDeviceDumpLocation(vth, dump_location) != 0) {
WARN("DISPLAY dump location failed, attempting SCALER...");
WARN("!! UNDOCUMENTED DUMP LOCATION !! OUTPUT MODE: 2...");
dump_location = 2;
if (DILE_VT_SetVideoFrameOutputDeviceDumpLocation(vth, dump_location) != 0) {
return -2;
}
}
Since i don't know if this change is working for everyone or just for me i didn't go ahead and make a pull request. i'll leave that up to you @Informatic, thanks for your hints!
We have done some enhancements in latest versions. Can you please check if your issue still applies?