An example usage of GraphicBuffer to work with GPU data
sergeivolodin opened this issue · 3 comments
I'm a developer at EPFL writing a library for augmented reality in education: https://github.com/chili-epfl/qml-ar which is supposed to run on Android.
At this moment we are trying to decrease the latency and the ability to read pixel data from GPU using a regular memcpy would be very beneficial for us.
I was trying to use this approach both on API 25 with a modified version of your library GraphicBuffer #5 and on API 27 using the public HardwareBuffer C++ wrapper.
However, in both cases, I've obtained only garbage output which looks suspiciously similar on different devices with different software and hardware.
I'm wondering if you have a working example of GLES calls which are required to obtain the pixels from the GPU, such as eglCreateImageKHR, glEGLImageTargetTexture2DOES, glFinish and the lock, unlock calls.
It seems that there are almost no successful stories on StackOverflow on this topic and I'm wondering if you have one.
If it would be possible to give such an example I believe the library could be very useful for many people since drastically decreasing latency in visual applications such as augmented reality is a very promising direction.
I think it would be beneficial to make GraphicBuffer run on all APIs 23-25-27 since we would like our library run on these and if I can verify that my fix for API 25 is correct, I could just do that since I have implemented both approaches already (except for the correct usage of the created GraphicBuffer).
Looking forward to any feedback and would be glad to know if there are any good examples of the library usage.
Sorry @sergeivolodin . I'm afraid I can't provide what you want. As you may have seen, it has been several years since the last commit. Actually this project was done when I worked for a start-up company as an intern 2 years ago, while currently I am not a mobile developer, even don't own an Android phone.
I'm very happy to guide people to your work once completed. Looking forward to it!
TLDR: The working example for APIs 26-27 and 24-25 is here: https://github.com/chili-epfl/GraphicBuffer (see README). Code from the original repo by @fuyufjh works for API 23.
I now have a working example for API27 (HardwareBuffer). Will try to make a working example using GraphicBuffer out of it
(.h) https://github.com/chili-epfl/qml-ar/blob/f5710ae90de4916f81f89cd016e2c9665fa3bf7c/src/imagebackend/nv21videofilterrunnable.h
(.cpp) https://github.com/chili-epfl/qml-ar/blob/f5710ae90de4916f81f89cd016e2c9665fa3bf7c/src/imagebackend/nv21videofilterrunnable.cpp
It's written in Qt, it uses NDK 16b. Tested with Qt 5.11.1 on Android 8.1.0 API 27
It currently does:
a) gets texture from QVideoFrame
b) on GPU using shaders: Downscaling + RGB->HSV + HSV thresholding (irrelevant w.r.t. GraphicBuffer)
c) Gets the resulting rendered image using EGLImage and wraps it as a QImage
I've noticed that it's important to attach the texture which is connected to EGLImage to a framebuffer as a color attachment.
I have tried glTexImage2D with some data on CPU (for testing) and glCopyTexImage2D (to copy data from a framebuffer), both resulted in black EGLImage
I will post here when I make a more user-friendly library, now I just post the code that worked for me.
The example is here in the README: https://github.com/chili-epfl/GraphicBuffer
Our team decided not to proceed with the library, so I leave the example here as it is.
It worked for API 25 (with hacks) and for API 27.