Terrain rendering issues with TAA/FSR (flickering)

[DevLewa]

Terrain3D version

v0.9.0

System information

Godot v4.2.1 - Windows 10, Forward+/Vulkan GTX 1060 6GB

Is the issue reproducable in the demo?

Not applicable

Issue description

Jittering/flickering can be seen/observed on the Terrain if TAA or FSR 2.0 is enabled. (With TAA it is less noticeable, with FSR its more pronounced)

Here a demonstration:
I set 3D scaling to FSR 2.0 and reduced the resolution scale to 0.1 (to make this really pronounced.)
https://www.youtube.com/watch?v=mKjPz8nHW2U

At first thought that this is a general Godot issue, howerver in the video only the terrain surfaces are flickering. (Other static geometry looks fine.)
I assume that this is due to the snapping/reposition behaviour of the 3D terrain renderer given that this flicker happens in a fixed intervall while moving. At least FSR to my knowledge requires a velocity buffer to reconstruct the image. The repositioning of the terrain might interfere with the FSR implementation casing incorrect reconstruction.

Not entirely sure how the velocity tracking in Godot is implemented. If we store the previous transform of the last frame (to calculate the delta for the velocity buffer each frame) one fix would be to artifically override the last transform state in case of a reposition to trick the image reconstruction on applying the correct velocity. (Have to check how Godot handles this internally.)

Can't test this with the demo project as of right now. (Having issues with godot not wanting to enable the addon on new projects due to parsing errors in the addons gd scripts. Will update this once i figure out what is going on.)

[TokisanGames]

Thanks for the report. I haven't used either yet. Is it the mesh or the textures flickering? Show it in wireframe mode, and checkerboard and grey debug views. Your trees flicker just as much.

[DevLewa]

Here a video with the default terrain grid texture and with enabled collision mesh:
https://www.youtube.com/watch?v=JJZRwbQwQiI

The trees flicker constantly (even stationary) as they have a material with alpha transparency + alpha clipping enabled. (without FSR it's already very aliased + i'm using a rather low resolution scale for demo purposes so it looks especially bad here.) However the terrain flickering shows a different behaviour. (It flickers only while moving in fixed intervals.)

You can also see that the grey grid texture on the cubes to the left are always stable while the floor is constantly jittering while moving. (it's stable if the camera doesn't change position in worldspace. Rotating the camera also works as expected.)
Hence my assumption that this is related to the velocity buffer/repositioning issue.

[InitialCon]

It's very likely that this is a motion vector problem. The height of the vertex is determined by its world position so when the terrain moves with the camera the position of the vertex will move and this will not write to the motion vector buffer. This is a problem since the fragment sample position moves too. Since TAA and FSR2 expect pixel colours to remain constant with a constant velocity it's reprojecting the previous sample from before the camera moved causing the blurring/flickering. I'm not sure if it's possible to fix this without outputting the velocity in the shader which is not currently supported. My TAA understanding isn't the best so I may be incorrect here or get some terminology wrong.

[TokisanGames]

I just tried FSR 1.0 and 2.2 in Godot 4.2.1 with the Terrain3D demo and they display perfectly fine on my NVidia 3070.

However when trying to load a project or a new terrain on my system it reports errors:

ERROR: Condition "p_job.uavMip[i] >= mip_slice_rids.size()" is true. Returning: FFX_ERROR_INVALID_ARGUMENT
   at: execute_gpu_job_compute_rd (servers/rendering/renderer_rd/effects/fsr2.cpp:418)

It also repeats these when moving the mouse around, and the decal mouse cursor doesn't show.

I just helped a user on discord who was having Godot crash with FSR 2.2 enabled on his NVidia 3080 while adding Terrain3D, with these errors:

ERROR: Width must be equal or greater than 1 for all textures
   at: (drivers/vulkan/rendering_device_vulkan.cpp:1722)
ERROR: Condition "texture.is_null()" is true. Returning: FFX_ERROR_BACKEND_API_ERROR
   at: create_resource_rd (servers/rendering/renderer_rd/effects/fsr2.cpp:240)

He probably needs to upgrade his drivers.

So, in conclusion, FSR 2.2 in Godot is not ready for production use. Even if it works for you, you have no idea if it will be stable on end user machines. We could possibly put some test conditions in Terrain3D to avoid problems if we know the triggers. However, the back end in the engine clearly has bugs which need to be identified, reported, and fixed on their end.

My recommendation is to not use it until it's much more stable.

[TokisanGames]

Regarding TAA: I do see occasional flickering in the demo in some sections of the rock texture, not all. It's minor. If I zoom up the textures so they're pixelated and set the filtering to nearest, it triggers TAA artifacts much more.

In Op's case, it looks like all of the textures are chosen specifically to conflict with TAA. If I put the texture on a mesh it looks fine. As @InitialCon wrote, the difference is that the terrain mesh moves around. Put the viewport in wireframe mode and you can see that when you laterally move the camera the mesh moves constantly. There's, very little TAA artifacts if the camera moves vertically only.

So TAA is likely not going to work. FSR seems fine in the demo, but maybe not with other textures, and is unstable anyway.

For now use MSAA or FXAA.