[REBLUR] Laggy reflections

Question

[REBLUR] Laggy reflections

xardas110 opened this issue 2 years ago · 17 comments

I am getting laggy reflections on my own project and on the NRD sample.
To reproduce this on the NRD sample: FPS cap at 30, No normal map and metallic at 1, gives lots of lag near the reflection as show in the video below:
https://user-images.githubusercontent.com/18483702/205492928-604dd7ac-17fb-4417-8477-76aa27ab5bf1.mp4

Is it supposed to be like this? How can one fix it in that case?

Answer 1 · 2022-12-05T09:16:08.000Z

@xardas110 Thanks for the report! There was a bug. I have just submitted a fix. The root cause was in imprecisely computed normal-based reprojection confidence, which uses normals from the previous frame stored with reduced precision. As the result for flat surfaces the confidence was close to 1 (but not 1) what should not have happened, because the fallback to surface-based motion is inapplicable in this case.

Before NY I will release a big update, which is already WIP. I have added the new test based on your description to avoid such regressions in the future.

Answer 2 · 2022-12-13T18:36:08.000Z

Thanks for the fast response. Been experimenting with it in the past days so sorry for the late response.
Your new update did fix the problem if the camera is near the wall with the reflection, but it also introduced the same problem when the camera is far away in my own project at least, hard to reproduce on the sample due to wall boundaries.
I will do more testing and see if I can fix it somehow, just in case I have done something wrong.
Other than that it seems like the reflections will always have some lag, if I am not mistaken?

Answer 3 · 2022-12-20T10:47:46.000Z

Please, show me a video. Please, also update to v4.0.0 when you have time. Several reprojection issues have been fixed in the latest version.

Answer 4 · 2022-12-31T07:34:56.000Z

So I have updated to the latest version(v4.0.1) and the only problem now with the reflections is lag on curved surfaces. Flat surfaces project the reflections without any lag now, but both in my own project and the NRD sample I have laggy reflections on curved surfaces, is this normal?

NRD sample(FPS cap 30), lag on curved reflective surface:
https://user-images.githubusercontent.com/18483702/210129002-4b83de65-3704-466f-aa72-0b4dff308df2.mp4

My sample, lag only happens when it's close to the reflective sphere:
https://user-images.githubusercontent.com/18483702/210129130-3cbc3f8a-0620-4137-9c93-5a8f703e490d.mp4

My sample, flat surface, no lag:
https://user-images.githubusercontent.com/18483702/210129695-713449cd-0464-45ce-ad6e-c09ab7bacc64.mp4

Answer 5 · 2023-01-04T20:08:34.000Z

NRD sample(FPS cap 30), lag on curved reflective surface...

Thanks for digging and checking the latest. Yes, it doesn't look correct, I will investigate.

Answer 6 · 2023-01-10T17:39:39.000Z

Observations:

lag in NRD sample in your video most likely related to wrong signs in thin lens equation in case of concave mirrors
lag in your sample can be related to imprecision of passed normals. Try with high quality normal encoding = 4, please.

Nevertheless I found several bugs and fixed them:

temporal stabilization pass now gets curvature in FP16, not a repacked one in 8-bit (important when the camera is very close to a large smooth object)
fixed race conditions while finding min hitT for specular tracking purposes (can be a reason of your problems too)
fixed signs of object and image distances in thin lens equation in case of concave mirrors (negative curvature)
improved curvature estimation in case of high parallax
dropped some hacks

I need to do more testing and then I will update GitHub (this week according to my plan).

Answer 7 · 2023-01-12T19:14:27.000Z

The video is running with normal encoding = 3, this was changed inside the cmakelists.txt inside the denoiser root folder, just mentioning it in case that's the incorrect way of doing it. I've tried 4 as well with no luck. The normal value also look best at encoding = 2, 3 and 4 gives less visible normals.

Answer 8 · 2023-01-12T19:33:09.000Z

It doesn't make a lot of sense, because 3 & 4 assumes 16-bit per normal component. OK, let's wait when I update GitHub (in a few days) to see the latest in action.

Answer 9 · 2023-01-16T11:25:16.000Z

@xardas110 Please, check v4.0.2.

Answer 10 · 2023-01-17T02:12:45.000Z

Same problem

Answer 11 · 2023-01-17T08:42:45.000Z

can the problem be reproduced in NRD sample?
if yes, does it happen on concave or convex mirrors?

Answer 12 · 2023-01-17T09:10:53.000Z

if yes, does it happen on concave or convex mirrors?

Can you add a test and send it to me, please? To achieve this:

unfold TESTS section
reproduce the problem
press add button
press on appeared test at the end of the list and ensure that visuals match your expectations
send me one of .bin files from Tests folder depending on the scene, you worked with

Answer 13 · 2023-01-18T06:15:03.000Z

Sorry for my vague reply earlier. I tested on my own project and it had the same problem, but with less lag. When testing on the NRD sample the concave mirror problem is gone, but I can still see ghosting on convex mirrors. Other than convex mirrors the denoiser seem to work flawlessly.

I added a few tests here for convex mirrors:
ShaderBalls.zip

Answer 14 · 2023-01-19T09:20:07.000Z

Thanks. I see the problem. OK, will do my best to fix it.

Answer 15 · 2023-02-19T06:54:51.000Z

I've somewhat found a Fix to this.

Answer 16 · 2023-09-21T02:23:04.000Z

It's worth trying the latest upcoming version (this week). But I know that REBLUR / RELAX don't reproject ideally on silhouettes of curved objects, when the camera is close to the object. I will keep this one open, because I'm very interested in proper fixing of this problem (by improving curvature math and reprojection, not by accelerating accumulation).

Answer 17 · 2023-12-27T04:10:41.000Z

I've somewhat found a Fix to this.

Closing then ;) Hopefully in the future I will be able to improve curvature in general.