About render the NV12 format
yuace opened this issue · 18 comments
void GL_VideoRenderer::update_texture_nv12(AVFrame* frame) {
assert(frame);
assert(frame->format==AV_PIX_FMT_NV12);
if(nv12_frametexture.textures[0]==0){
std::cout<<"Creating nv12 textures\n";
glGenTextures(2, nv12_frametexture.textures);
}
int videoWidth = frame->width;
int videoHeight = frame->height;
int size = av_image_get_buffer_size((enum AVPixelFormat)frame->format, frame->width,frame->height, 1);
static uint8_t * out_buffer = (uint8_t*)av_malloc(size);
av_image_copy_to_buffer(out_buffer, size,
(const uint8_t * const *)frame->data,
(const int *)frame->linesize, (enum AVPixelFormat)frame->format,
frame->width, frame->height, 1);
for(int i=0;i<2;i++){
glBindTexture(GL_TEXTURE_2D,nv12_frametexture.textures[i]);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE, frame->width,frame->height>>i, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, out_buffer+videoWidth*videoHeight*i);
glBindTexture(GL_TEXTURE_2D,0);
}
nv12_frametexture.has_valid_image= true;
av_frame_free(&frame);
}
I decode the format is nv12, so I refer to the cuda. But the display seems like just have the Y, the uv is not correct.Could u give some advice about it?
has done, I use rgb can work, but latency seems like very high.
Looks like some stride incompability, or some bugged graphics driver. What's your system though ? This / these paths are for x86 (where you can cpu copy a whole frame okay) only.
You can try going for the egl texture path if you are on embedded. Not ideal, but lower latency than cpu copy.
Looks like some stride incompability, or some bugged graphics driver. What's your system though ? This / these paths are for x86 (where you can cpu copy a whole frame okay) only.
1.I check the vpu decode the video is correct,but if use the YCbCr,the uv not correct, so i change the shader to rgb:
"vec3 yuv;\n"
"vec3 rgb;\n"
" yuv.r = texture2D(s_texture_y, v_texCoord).r;\n"
" yuv.g = texture2D(s_texture_uv,v_texCoord).r-0.5;\n"
" yuv.b = texture2D(s_texture_uv,v_texCoord).a-0.5;\n"
" rgb = mat3(\n"
" 1.0f, 1.0f, 1.0f,\n"
" 0.0f, -0.39465f, 2.02311f,\n"
" 1.13983f, -0.5806f, 0.0f"
" )*yuv;\n"
" gl_FragColor = vec4(rgb, 1.0);\n"
"}\n";
2.You mean now the high latency is from cpu copy a whole frame(av_image_copy_to_buffer)?But cuda seems like the same(copyCudaFrameToTextures(frame)).
3.I use arm64(qualcomm qrb5165), after I get the avframe from the vpu(AV_PIX_FMT_NV12), egl path can try,but not ideal,right? So if there have the common nv12 texture path?
If you are on Qualcomm (which is pretty much a phone soc) I'd go for egl. Then you save the CPU copy step to the gpu. Latency then depends on more factors, but egl, that's what I'd do first.
Egl also does the nasty yuv conversion crap for you
"Egl texture external
That's great, let me try it, thanks a lot.
egl texture external, you base on AV_PIX_FMT_DRM_PRIME?
AV_PIX_FMT_DRM_PRIME and AV_PIX_FMT_NV12 the avframe data is same?
No, for egl external, you need AV_PIX_FMT_DRM_PRIME (or whatever extension qualcom provides).
It is really popular on android, so I'd expect qualcom to provide the proper tool on linux, too.
On rpi, we are already using it for h265, so most of the programming work is already done (plugging things on platform X then can be tricky of course).
"since the frame data needs to be decoded into a GPU buffer (or memory accessible by GPU)
DRM_PRIME does the bridge (and if data in the buffer is then yuv420, nv12, whatever, you don't need to care - it is wired up by the vendor in the egl external shader)
Yes, I want to look for the method, qualcomm seems like privode the GBM, zero copy to display.
@yuace I also use the nv12 pixel format to display. I have already changed the shader draw code as follwing:
static const GLchar* fragment_shader_source_NV12 =
"#version 100\n"
"precision highp float;\n"
"uniform sampler2D s_texture_y;\n"
"uniform sampler2D s_texture_uv;\n"
"varying vec2 v_texCoord;\n"
"void main() { \n"
" const vec3 offset = vec3(-0.062745, -0.501960814, -0.501960814);\n"
" vec3 YCbCr; \n"
" YCbCr.x = texture2D(s_texture_y, v_texCoord).r; \n"
" YCbCr.y = texture2D(s_texture_uv,v_texCoord).r; \n"
" YCbCr.z = texture2D(s_texture_uv,v_texCoord).g; \n"
" YCbCr += offset;\n"
" mat3 colorMatrix = mat3(\n"
" 1.1644f, 1.1644f, 1.1644f,\n"
" 0.0f, -0.3917f, 2.0172f,\n"
" 1.5960f, -0.8129f, 0.0f"
" );\n"
" gl_FragColor = vec4(colorMatrix * YCbCr, 1.0);\n"
"}\n";
The issue seems about the uv texture value.
In gl_videorenderer.cpp source:
glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE, widths[i], heights[i], 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, frame->data[i]);the GL_LUMINANCE means texture2D() r,g,b have same value. So can you give me some suggestion on how to display NV12 frame correct.