Native android application that showcases camera preview mapping on a spinning 3D cube.
Native android application developed with the use of Vulkan API that performs real-time camera preview mapping on a spinning 3D cube. The aim of the project is to provide with an elaborate example that showcases various useful, according to the author, basic-to-intermediate developing techniques for the creation of Vulkan-powered native android applications that make use of multiple hardware facilities.
This side-project started as an effort to refresh and update my C++ and 3D graphics programming skills. The initial purpose was the study of the latest C++ standards (C++17 and "C++20") as well as the study of the Vulkan library. After an introductory review, it became apparent that Vulkan is mainly used by C rather than C++ developers although the Vulkan C++ library (https://github.com/KhronosGroup/Vulkan-Hpp) is constantly gaining popularity. Also the latest developments on mobile graphics and camera hardware along the fact that Vulkan runs on Android devices, drove me down the path of exploring the possibilities of building system applications for Android (outside the JVM context) in order to take advantage of the Vulkan graphics and compute capabilities as well as other third-party system libraries (e.g. OpenCV, dlib, TensorFlow) for my domains of interest. These domains being Computer Vision, Computer Graphics and Augmented Reality.
By further studying and working on this idea, I also discovered that there aren't much resources out there on how to integrate the camera into a 3D native Android application using "immediate memory mapping" (external buffers). So I've decided to build a single-threaded (well if you don't count the JVM context threads, and the ones related to the hardware devices) Vulkan sample that showcases the following:
-
Usage of C++17, Vulkan Hpp and the latest NDK (see Pre-requisites) for building a native Android 3D application.
-
Management of Android user permissions outside the JVM context (i.e. without the use of JNI calls).
-
Design of a clean native program loop that manages user input, sensors (e.g. accelerometer, camera) and screen output.
-
Design of a basic Vulkan context for 3D rendering.
-
Camera-Vulkan APIs interoperability: Communication of camera data, via the usage of Camera2 API and utilization of external hardware buffers, to the Vulkan context.
In the references section, I list all sources that guided me in realizing this project.
The style and conventions of the project's codebase targets intermediate-level C++ programmers with some knowledge of the C++17 standard. The project is made in Android Studio with the aid of CMake. Thus, knowledge of this IDE and building tools (CMake, Gradle, AVD & SDK Managers) is required. The development OS was Windows 10. Finally, native-camera-vulkan, makes use of some external to NDK binary and header libraries, so the developer must configure the development environment in such a way (acquiring needed libraries, configure locations etc.), as to meet the following.
| Development Tools | |
|---|---|
| Android Studio | v4.1.2 |
| NDK | v22.0.7026061 |
| SDK Platform | v30.3 |
| SDK Build-Tools | v30.0.3 |
| External (to AS) Development Tools | |
| CMake | v3.19.2 |
| External (to NDK) Libraries | |
| Vulkan | v1.2.162 |
| Vulkan HPP | v1.2.162 |
| Validation Layers | v1.2.162 |
| GLM | v0.9.9.8 |
| STB | v2.26 (stb_image) |
For more information see Vulkan C++ API and NDK and Build Instructions.
native-camera-vulkan is built to support devices running Android Oreo (v8.0) or newer (minimum SDK Version is 26), but... Some strong assumptions have been made here and there throughout the project. The application expects from the device, to support certain Vulkan extensions. Not all devices could run this application, but the ones that support the following (in addition to supporting running Vulkan, that is 😃):
| Instance Extensions |
|---|
| VK_KHR_surface |
| VK_KHR_android_surface |
| VK_KHR_external_memory_capabilities |
| VK_KHR_external_semaphore_capabilities |
| VK_KHR_get_physical_device_properties2 |
| VK_EXT_debug_utils |
| Logical Device Extensions |
| VK_KHR_swapchain |
| VK_KHR_maintenance1 |
| VK_KHR_bind_memory2 |
| VK_KHR_get_memory_requirements2 |
| VK_KHR_sampler_ycbcr_conversion |
| VK_EXT_queue_family_foreign |
| VK_KHR_external_memory |
| VK_KHR_external_semaphore |
| VK_KHR_external_semaphore_fd |
| VK_KHR_dedicated_allocation |
| VK_ANDROID_external_memory_android_hardware_buffer |
Also it is expected that the device has a front and a back camera. It is fixed for the back facing camera to be selected for usage (this can be changed through the code here). The application was tested successfully on a Nokia 6.1 and a Samsung Galaxy A50. For the development environment a Nvidia GTX 770 was used.
The Android NDK used in this project (see Pre-requisites), ships with an outdated version of the Vulkan headers (v1.2.121). These headers do not correspond to a Vulkan HPP library version (at least not an interesting one). So it is decided that for the purposes of the development, a newer version of the libraries as well as of the validation layers will be used (v1.2.162). In order to do so, besides including the newer headers in the project, the corresponding binaries of the validation layers had to also be introduced as dependencies in Gradle, see Build-Instructions for more details.
The Android SDK provides the ability, to the developer, to build a native Android application without writing any Java code at all. One way of doing this, is by leveraging the fact that a Java NativeActivity can be defined implicitly via the AndroidManifest.xml in combination to making use of the android_native_app_glue.h helper library on the C++ side. In this way, the execution process creates a Java context (JavaVM) from which spawns a separate thread to be used as a starting point for native event loop handling, native procedures etc. (using the helper library) and all this are abstracted by the developer.
It is a common practice in native Android applications to use Java code for handling basic stuff, such as app lifecycle management, app permissions management, event management etc. and utilize Java Native Interface (i.e. JNI) to invoke native procedures. In this project the native code manages the application events by itself and also uses JNI in a reverse manner, from C++ to Java, in order to invoke Android Java facilities not available otherwise in the native side (e.g. app permissions management).
The communication of camera images to the graphics engine is made via the usage of AHardwareBuffer (see Native Hardware Buffer) objects. Camera device is set to work at a Preview mode in order for frames to captured as fast as possible (this doesn't favour quality, for more information see here). A simple synchronous image reader is implemented and governed by the application engine. This means that frames are requested at a rate that corresponds to the rendering frequency (i.e. as fast as possible). Typically, a camera device would be much slower than the graphics engine. For this, the image reader also plays another role. It decouples the camera device from the graphics engine so that the graphics engine can request frames as fast as it can without the burden of handling unanswered requests, while the camera device can serve frames at it's own pace. Off course, in the event of absence of a new image, graphics engine renders the old one(s).
Touchscreen (i.e. input) and accelerometer devices have also been integrated into the application. Any change of touch input (e.g. touch screen coordinates) or of the orientation of the physical device is reflected in the application as background color change.
Special care has been given to logging formatting. A custom logging library that wraps android logging facilities is implemented. The custom library produces among others, tabular output like the one presented below:
INFO: Version is: 0.2.0-1127 (build.1883)
DEBUG: Generic engine has been initialized.
INFO: Vulkan library loaded with success.
DEBUG: ┌---------------------------------------------------------------------┐
DEBUG: | GENERAL VULKAN INFORMATION |
DEBUG: |---------------------------------------------------------------------|
DEBUG: | Available Extensions |
DEBUG: | ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ |
DEBUG: | VK_EXT_debug_report v0.0.9 |
DEBUG: | VK_EXT_swapchain_colorspace v0.0.4 |
DEBUG: | VK_KHR_android_surface v0.0.6 |
DEBUG: | VK_KHR_external_fence_capabilities v0.0.1 |
DEBUG: | VK_KHR_external_memory_capabilities v0.0.1 |
DEBUG: | VK_KHR_external_semaphore_capabilities v0.0.1 |
DEBUG: | VK_KHR_get_physical_device_properties2 v0.0.2 |
DEBUG: | VK_KHR_get_surface_capabilities2 v0.0.1 |
DEBUG: | VK_KHR_surface v0.0.25 |
DEBUG: |---------------------------------------------------------------------|
DEBUG: | Available Layers |
DEBUG: | ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ |
DEBUG: | VK_LAYER_KHRONOS_validation |
DEBUG: | VK_EXT_debug_report v0.0.9 |
DEBUG: | VK_EXT_debug_utils v0.0.2 |
DEBUG: └---------------------------------------------------------------------┘
INFO: Found a Vulkan device. Context created.
...
With the combination of an appropriate filter in logcat panel of Android Studio, one can greatly improve online logging inspection of the running application.
For a successful build, v3.19.2 of CMake must be installed in the development system and registered in the development environment via PATH variable. The project is configured to look for an externally managed (in respect to the Android Studio) cmake binary.
Also LIBRARIES_ROOT environment variable must point to a location where the dependencies listed in Pre-requisites exist in a pre-specified directory structure as detailed in the table below:
| Local Directory | Mapped Source |
|---|---|
| %LIBRARIES_ROOT%\vulkan\vulkan | Repository\include\vulkan |
| %LIBRARIES_ROOT%\vulkan_hpp\vulkan_hpp | Repository\vulkan |
| %LIBRARIES_ROOT%\stb\stb | Repository\stb1 |
| %LIBRARIES_ROOT%\glm\glm | Repository\glm\glm |
| %LIBRARIES_ROOT%\vulkan_validation_layers\bin\android-1.2.162 | (extracted layer binaries .zip) |
The provided cmake configuration is parameterized. The developer can control the build output by configuring two compilation aspects.
First, one can configure BUILD_FLAVOR variable in order to determine whether to build a simplified version of the application (Simple Vulkan Context) or the full (Complex Vulkan Context) version.
Second, by tweaking on and off certain compilation flags one can select whether to include or not in the final binary, some of the following:
- Debug Build Indication
- Validation Layer
- Logging Facility
- Profiling Facility
In total, 8 different build outputs can be produced by the various compilation configurations that can be specified. The following table provides an overview:
| CMAKE Variable | |
|---|---|
| BUILD_FLAVOR | SIMPLE_VULKAN: Basic Vulkan context, no graphics pipeline |
| COMPLEX_VULKAN: Complete Vulkan context | |
| Compilation Flags | |
| NDEBUG | Set for release builds |
| NCV_VULKAN_VALIDATION_ENABLED | Enable validation layer |
| NCV_LOGGING_ENABLED | Enable custom logging facility |
| NCV_PROFILING_ENABLED | Enable profiling facility |
- A Tour of C++, 2nd Edition - Bjarne Stroustrup: Quick overview of C++17 and a general not-in-depth C++ refresher for system programmers.
- Vulkan Specification: The best Vulkan resource.
- NDK Guides: The best native Android development resource.
- Camera2 API Reference.
- Vulkan Samples: Official Vulkan samples repository.
- Vulkan C++ Examples and Demos: Awesome repository with a great number of helpful samples.
- Vulkan Tutorial: Great resource for Vulkan beginners.
Contributions are always welcome! I'll try to review new pull requests frequently, make sure to include a clear and detailed summary about the proposed changes. Also feel free to contact me for any questions, in any of the advertised ways in my profile.
If you found this work helpful and want to support the author, consider making a donation.
This project is licensed under the general terms of the Apache-2.0 License. See LICENSE for more information related to the usage and redistribution of this project.
- Vulkan and Vulkan logo are registered trademarks of the Khronos Group Inc.
- Android and Android logo are registered trademarks of the Google LLC.
- C++ logo is registered trademark of the Standard C++ Foundation.