Vulkan glTF Scene Raytrace/Raster

Pathtracer	Raster

This sample loads glTF (.gltf/.glb) scenes and will ray trace or rasterize it using glTF 2.0 material and textures. It can display an HDR image in the background and be lit by that HDR or use a built-in Sun&Sky. It renders in multiple passes, background, scene, and then tone maps the result. It shows how multiple resources (geometry, materials and textures) can be shared between the two rendering systems.

Pathtracer

Implements a path tracer with global illumination.

The options are:

Max Depth : number of bounces the path can do
Max Samples: how many samples per pixel at each frame iteration
Debug Method: shows information like base color, metallic, roughness, and some attributes

Raster

The rasterizer uses the same Vulkan resources as the path tracer; scene geometry, scene data, textures. And for shading, it shares many of the same functions.

The options are:

Show wireframe: display wireframe on top of the geometry
Debug Method: shows information like base color, metallic, roughness, and some attributes

Example with wireframe option turned on

Debug

There is also the ability to debug various out channels, such as:

metallic	roughness	normal	base	emissive

Environment

It is possible to modify the environment, either by choosing an integrated sun and sky, or by lighting the scene using Image Base Lighting. In the latter case, you need an image (.hdr). You can find examples of such images at the following address Poly Haven

Sun & Sky	HDRi

Having HDRi (High Dynamic Range Imaging) to illuminate the scene greatly simplifies complex lighting environments. It also helps to integrate 3D objects into its environment.

This example loads HDR images, then creates an importance sampling acceleration structure used by the ray tracer and stores the PDF in the alpha channel of the RGBA32F image.

For real-time rendering, we use the created acceleration structure and create two cubemaps. One containing the diffuse irradiance and the other, storing the glossy reflection, where the different levels of glossiness are stored in separate mipmap levels.

Tonemapper

We could not get good results without a tone mapper. This is done with a compute shader and different settings can be made.

Multiple tonemapper are supported:

Filmic
Uncharted 2
Clip : Simple Gamma correction (linear to sRGB)
ACES: Academy Color Encoding System
AgX
Khronos PBR : PBR Neutral Specification

Camera

The camera navigation follows the Softimage default behavior. This means, the camera is always looking at a point of interest and orbit around it.

Here are the default navigations:

The camera information can be fine tune by editing its values.

Note: copy will copy in text the camera in the clipboard, and pressing the paste button will parse the clipboard to set the camera.

Ex: {0.47115, 0.32620, 0.52345}, {-0.02504, -0.12452, 0.03690}, {0.00000, 1.00000, 0.00000}

Save and Restore Cameras

It is also possible to save and restore multiple cameras in the second tab. Press the + button to save a camera, the middle button to delete it. By pressing one of the saved cameras, its position, interests, orientation and FOV will be changed smoothly.

Note: If the glTF scene contains multiple cameras, they will be showing here.

Other modes

Other navigation modes also exist, like fly, where the w, a, s, d keys also moves the camera.

Schema of the Program

The nvvk::Application is a class that provides a framework for creating Vulkan applications. It encapsulates the Vulkan instance, device, and surface creation, as well as window management and event handling.

When using nvvk::Application, you can attach nvvkhl::IAppElement to it and each element will be called for the different state, allowing to customize the behavior of your application. The nvvkhl::IAppElement class provides default implementations for these functions, so you only need to override the ones you need.

Here is a brief overview of how nvvk::Application works:

Initialization:

When you create an instance of nvvk::Application, it sets up the Vulkan instance, device, and surface. It also creates a window and sets up event handling.

Attaching Elements

In main() we are attaching many elements, like:

ElementCamera : this allow to control a singleton camera
ElementProfiler : allow to time the execution on the GPU
ElementBenchmarkParameters : command line arguments and test purpose
ElementLogger : redirect log information in a window
ElementNvml : shows the status of the GPU

But the main one that interest us, and which is the main of this application is GltfRendererElement. This is the one that will be controlling the scene and rendering.

Main Loop:

The nvvk::Application class provides a main loop that continuously processes events and updates the application state. Inside the main loop, it calls the following functions:

onAttach():
This function is called whenever the element is attached to the application. In GltfRendererElement, we are creating the resource needed internally.
onDetach():
This function is called when the user tries to close the window. You can override this function to handle window close events.
onRender(VkCommandBuffer):
This function is called to render the frame using the current command buffer of the frame. You can override this function to perform rendering operations using Vulkan. In GltfRendererElement this is where the active renderer is called.
onResize():
This function is called when the viewport is resized. You can override this function to handle window resize events. In GltfRendererElement the G-Buffer will be re-created
onUIRender():
This function is called to allow the IAppElement to render the UI and to query any mouse or keyboard event. In GltfRendererElement, we render the UI, but also the final image. The rendered image is consider a UI element, and that image covers the entire viewport ImGui window.
onUIMenu()
Will be modifying what we see in the the window title. It will also create the menu, like File, Help and deal with some key combinations.
onFileDrop()
Will receive the path of the file been dropped on. If it is a .gltf, .glb or .hdr, it will load that file.

glTF Core features

What is currently not supported are animations and skins, multiple textures coordinates, morph targets, color at vertices, and some extensions.

GLTF Extensions

Here are the list of extensions that are supported by this application

Scene Graph

The GLTF scene is loaded using tinygltf and then converted to a Vulkan version. The Vulkan version is a simplified version of the scene, where the geometry is stored in buffers, and the textures are uploaded to the GPU. The Vulkan version is used for both raster and ray tracing.

The scene is composed of nodes, where each node can have children and each node can have a mesh. The mesh is composed of primitives, where each primitive has a material. The material is composed of textures and parameters. However, none of this is directly used in the rendering, as we are using a simplified version of the scene.

Once the scene has been loaded, we proceed to parse it in order to collect the RenderNodes and RenderPrimitives. The RenderNode represents the flattened version of the tree of nodes, where the world transformation matrix and the material are stored. The RenderPrimitive, in contrast, represents the unique version of the primitive, where the index and vertex buffers are stored.

RenderNodes represent the elements to be rendered, while RenderPrimitives serve as references to the data utilized for rendering.

Animation

If there is animation in the scene, a new section will appear under the Scene section. It allows to play/pause, step and reset the animation, as well as changing its speed.

Multiple Scene

If there are multiple scenes, a new section will appear under the Scene section. It will show all the scenes and their name. Clicking on a scene name will switch to the scene.

Material Variant

If there are multiple material variant, a new section will appear under the Scene section. It will show all the material variant and their name. Clicking on a variant name will apply it on the models.

Scene Graph UI

It is possible to visualize the scene hierarchy, to select node, to modify their transformation and their material, to some level.

nvpro-samples/vk_shaded_gltfscene