[PERF] SteamVR reprojects the wrong frames

Question

[PERF] SteamVR reprojects the wrong frames

Opened this issue 5 years ago · 22 comments

Zamundaaa commented 5 years ago

System Information (please complete the following information):

Distribution: Manjaro KDE
SteamVR version: 1.8.20
Steam client version: 1573176642
Opted into Steam client beta?: Yes
Graphics driver version: Mesa 20.0.0-devel (git-2e64afa5e3) (ACO)
Gist for SteamVR System Information: https://gist.github.com/Zamundaaa/b8f038e4f966ebaf76f97b4237c933a9

Performance Data (optional, but very helpful):

CPU: i7 6700k
GPU: rx 5700 XT
gists for perf top are below
couldn't use gpuvis, nothing happened upon pressing the button in debug commands.

kill -10 $(pgrep -f gpu-trace)

just returns

kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

Describe the bug

There is quite often noticable lags where it seems like the view is flickering to another orientation or something like that.
The specific issue can be seen in this (it ran very bad for some reason there) frame timing graph from Sairento VR: Either the frame timing graph doesn't work as intended or SteamVR sometimes reprojects in frames where it shouldn't and doesn't reproject in frames where it definitely should.
I used Sairento VR as the example here as it apparently triggers the issue the most but it's also observable in Blade&Sorcery, SteamVR Home etc, probably in any VR application.

perf top async reprojection
I'm not quite sure if the graph is this spikey because of how Sairento renders or if it's (partially) caused by SteamVR, but the reprojection behaviour is definitely flawed.

Setting "use legacy reprojection mode" in the Applications tab disables all reprojection alltogether, there's always stuttering and it's just horrible:

sometimes even like this:

perf top legacy reprojection

While writing this, SteamVR started going into a state where the frame timing graph looks like this (Sairento VR still running):

It didn't seem to get out of this state, it only got worse:

After a while it began looking like this:

The advanced frame timing graph for this looks like that:

One more thing perhaps worth noting: When (using CoreCtrl) setting GPU and memory clocks to the default maximum and the power profile to "Virtual Reality" instead of automatic the situation almost always improved.
All the tests I did was on 100% resolution @ 90fps (recommended setting: 122%) with Proton 4.11-8.

Note: Commenters who are also experiencing this issue are encouraged to include the "System Information" and "Perf Data" sections in their replies.

Answer 1 · 2019-11-26T17:57:56.000Z

Just to make sure it's not a problem with Proton I tried The Talos Principle VR, and it's even worse there... Although the frame timing graph doesn't necessarily look much worse.

I also tested Beat Saber:

and Blade & Sorcery:

All games exhibit this behaviour for me, the difference is only in how bad it gets.
The tests I did was again at 100% resolution @ 90fps, Proton 4.11-8 (except for The Talos Principle VR of course) and with the launch option "RADV_PERFTEST=aco %command%", including SteamVR itself. Using LLVM for SteamVR doesn't seem to make a noticable difference with the resoluton manually set to 100%.

Answer 2 · 2019-12-11T14:42:34.000Z

Is there no fix in sight or more data to provide? The stuttering because of this is quite headache inducing for longer play sessions.

Answer 3 · 2020-01-05T23:08:23.000Z

Also experiencing general performance issues with SteamVR on Linux despite Windows generally performing better/smoother using the same hardware. 😞

Redout (Ubuntu 18.04 LTS, DXVK 1.5)

Project CARS 2 (Ubuntu 18.04 LTS, DXVK)

Project CARS 2 (Windows 10, DX11)

It appears Motion Smoothing is also required for other games to reach performance targets, any reason why it's unavailable for NVIDIA users on Linux?

No Man's Sky (Windows 10, Vulkan)

The Forest (Windows 10, DX11)

System Information:

Distribution: Ubuntu 18.04 LTS, Windows 10
SteamVR version: 1.9.16
Proton version: 4.11-11
Steam client version: Ubuntu, Windows
Opted into Steam client beta?: Yes (Ubuntu), No (Windows)
Graphics: RTX 2060
Graphics driver version: NVIDIA 440.44 (Ubuntu), NVIDIA 441.66 (Windows)
Gist for SteamVR System Information: Ubuntu, Windows

Note: Commenters who are also experiencing this issue are encouraged to include the "System Information" section in their replies.

Answer 4 · 2020-01-06T11:41:38.000Z

A big part of the reason that you're having problems with performance is likely that there's no Async Reprojection for Linux users on NVidia and no Motion Smoothing on Linux in general.
That's not really related to this issue (this issue is mostly about async reprojection reprojecting the wrong frames and legacy reprojection not doing anything at all, although I do have to admit that the tite isn't exactly specific either);
From those frame time graphs you show it seems like reprojection is actually (at least ignoring the Project Cars 2 graph, idk what's going on there) working fine for you.

Answer 5 · 2020-02-17T19:19:05.000Z

So I downloaded gpuvis, compiled it, installed trace-cmd-git from the AUR and calling it manually works!
SteamVR Home (edit: async reprojection was disabled): https://drive.google.com/open?id=1vb3dVz4YxkKJUSTQsKHsDFHE1XYWZhyx
Blade and Sorcery: https://drive.google.com/open?id=1Em31FusX_ppRjPmBZPTjrF_6swMuqRgc
Blade and Sorcery without async reprojection: https://drive.google.com/open?id=1zyQ-5wBCjQs6Dunw3agHOBwMW4mPAzOG

I'm not entirely sure but it does really look like SteamVR is more or less completely ignoring the vblanks, running faster than 90Hz, and that's what creates the stuttering.
@lostgoat could this be some kind of problem with botched timings again?

Answer 6 · 2020-02-19T17:16:57.000Z

Perhaps also of use: without any apps open at all
Exactly the same problem, but there's only a dropped frame every few seconds or so and no reprojection at all.

Answer 7 · 2020-03-01T16:06:51.000Z

I don't have Windows to compare, but I also experience this issue with my Vive and would love to see progress on this.

Unfortunately I'm quite the weakling when it comes to motion sickness, and anything that messes up performance does not help :/

Answer 8 · 2020-03-21T01:12:02.000Z

The newest update (1.10.32) has some changes on the reprojection behavior.
From what I could see the algorithm is now very slow at switching between full speed and reprojection, and it seems to really favor to reproject rather than not. For example: it didn't stop reprojecting at 7ms frame times at 120 Hz (8.3ms).
It feels like some bad stuttering / flickering got resolved because of less enabling and disabling of reprojection (yay!) but the actual problem is still there, it reprojects in the wrong frames. Even while the algorithm decided to always reproject there were still single frames that either correctly didn't reproject or that were actually spikes and actually should've been reprojected...

Answer 9 · 2021-08-28T22:05:54.000Z

It seems like not only async reprojection is affected after all. I put

"enableLinuxVulkanAsync" : false

into steamvr.vrsettings in hopes of not having to enable the "use legacy reprojection" thing in the settings for every app. With that async reprojection is visibly disabled but it still reprojects the wrong frames - only by using "legacy reprojection", which seems to just not do any reprojection at all this bug can be worked around (as long as games keep up of course, when they don't it's even worse than before)

Answer 10 · 2021-09-25T12:28:31.000Z

[BUG] Async frame hallucinations OVER & UNDERshoot rotation vector below 60fps (video)

Issue transferred from #467.
@SpookySkeletons posted on 2021-09-25T02:16:14:

The bug is present on both AMD and Nvidia hardware as soon as async motion smoothing is enabled.
The issue becomes apparent below 60 fps.

Improperly hallucinated frames are visible flickering both in the direction of movement as well as trailing behind, alternating seemingly at random.

Distribution: [Gentoo]
SteamVR version: [E.g. 1.20.2]
Steam client version: [Sep 17, 2021]
Opted into Steam client beta?: [Yes]
Graphics driver version: [run nvidia-settings or vulkaninfo | grep driverInfo: nvidia RTX 2070 470.63.01
Gist for SteamVR System Information: [See instructions]

VIDEO
https://www.youtube.com/watch?v=JMNXO1_dtHs
Video is poor quality phone within HMD but gets the point across, use the comma and period to navigate frame by frame, you can see the rapid jittering motion of the hallucinated frames.

Note: Commenters who are also experiencing this issue are encouraged to include the "System Information" section in their replies.

Answer 11 · 2021-10-06T09:13:36.000Z

@kisak-valve
https://steamcommunity.com/app/250820/discussions/3/2564160288796057792/
Seems like this issue has come up for other platform drivers. Looked to be a scale mismatch on the motion vectors on the early release of Windows Navi GPU drivers. Are you able to fiddle with the value and see if it makes a difference?

Answer 12 · 2021-12-12T22:03:18.000Z

@kisak-valve
Did some closer inspection of VR compositors in general and how OpenVR seems to handle.
Think I may have found what is going on.
Your Async timewarp works, by halving the framerate and then panning the projection by a degree to match up with the new look vector (45-90 fps looks fine).
Your async spacewarp is broken, it either isn't sampling motion vectors when it attempts to reproject the old or the new frame creating all or nothing dead frames that fully diverge from where the estimation should actually be (below 45 fps actually introduces more stutter than it solves scaling the lower you fall). Causes double vision & jerking stutter.

Please check your motion estimation or motion sample application on linux. In the mean time please allow the user to specifically disable async spacewarp in the config until you can fix the issue.

Answer 13 · 2021-12-13T14:40:50.000Z

@SpookySkeletons please scroll up, the problem is already well specified and you can disable async reprojection.

Answer 14 · 2022-03-21T17:03:23.000Z

I'm curious if this may have something to do with it, anybody tried kernel 5.17?: torvalds/linux@8590222

The logic would be something like: If the HID-devices (HMD, controllers, trackers) are being read sequentially instead of in parallel, then sometimes the position-read which the async reprojection code gets from the headset will be behind its actual position (when the HMD can't be polled because one of the other devices are busy with the hidraw table), leading to a reprojected frame that is where it was earlier. Then, after that, async repro gets an updated (correct) position from the HMD, extrapolates between the wrong frame and the correct frame, and overshoots the next frame.

Answer 15 · 2022-05-20T03:20:22.000Z

Tested with kernel 5.17.8 (Mainline code + Debian Config, adjusted 250->1000Hz), Mesa 21.3.8, SteamVR 1.22.12. Still an issue.

Ryzen 3900X, 64GB RAM, 6700XT

GPU in the VR power profile to ensure it doesn't dynamically downclock.

Even just in the empty environment (eg start SteamVR with SteamVR Home disabled, close the dashboard) spinning back and forth there's misprojected frames (despite frametimes being ~1.4ms, so no reason for any reprojection).

In VRChat, there's visual corruption when in the "empty environment" (when the game stops rendering frames for a second or two) as well as the blue inbetween screen that SteamVR renders (The "Initializing World" blue screen)

I'm going to test under Wayland, seeing as someone worked around #499.

EDIT: Unfortunately, I could not get the workaround to work, so I was unable to successfully launch SteamVR in Wayland with async reprojection.

Answer 16 · 2022-05-24T17:01:46.000Z

Wayland is exactly the same.
This issue is a matter of the sampled fill in frames not actually sampling any motion vectors and going all-or-none on their presentation.
Looks like we're going to have to NOOP some of the async hallucination functions in steamvr's compositor binary to prevent it entirely from generating or presenting these vector sampled smear frames as "disabling async" does not actually prevent this path from running.
Any Ghidra wizzes would be appreciated if I cant find the time (or skill).

Answer 17 · 2023-05-26T23:36:37.000Z

Once again, the radio silence on valve's part is stunning.

Answer 18 · 2023-07-19T18:21:42.000Z

While messing around with wlxoverlay I ran into this on their discord:

lightwo
BOT
— 07/16/2023 8:32 PM
just wanted to point out I figured out why I had reprojection and performance issues with SteamVR on AMD, apparently everything works 10x better if games are forced to use AMDVLK while SteamVR is forced to use RADV

not my discovery, I found out here, but it makes all the difference in the world
https://steamcommunity.com/sharedfiles/filedetails/?id=2903413714
Steam Community :: Guide :: Linux AMD GPUs - How to get good perfor...
With three commands and some launch options, fix your HL:A issues on AMD GPUs....
Steam Community :: Guide :: Linux AMD GPUs - How to get good perfor...
not all games work this way, Neos for example becomes a bunch of artifacts, however I found that simply installing AMDVLK and forcing the game to use RADV still fixed reprojection in newer versions

Which leads to https://steamcommunity.com/sharedfiles/filedetails/?id=2903413714

I haven't seen it mentioned anywhere around here, so I thought it might be worth posting.

Edit: So to put this in laymans terms (which is sadly the only terms I'm capable of :), if the above is to be believed, the root cause of the reprojection-misery is that the SteamVR overlay (dashboard) is fighting for control of the icd-loader with whatever app is running (SteamVR Home for instance), and the race between the two is what leads to frames overshooting/undershooting the frames in the running app when reprojection kicks in. Reciprocally this could feasably also be the root cause of the wobbly windows-bug #395 as that would be the smooth version of this reprojection-race that affects the SteamVR overlay/dashboard.

Anecdotally this idea that there are two reprojection scenarios vying for control fits pretty well with my own experience in VRChat: If I go to a taxing world with many people that really stresses my system, reprojection kicks in in the app with it's overshot frames and stutter to the point of unplayability; but if I then bring up the dashboard, lo and behold, everything going on in the background behind the dashboard is suddenly moving buttery smooth even though there's technically more on screen now than before.

I'll try messing around with the lightwo's solution later today, see if I can get some numbers on it. Sadly, it's not a "fix"-fix if it works, just a workaround,, but it might point someone more capable in the direction of a more permanent solution.

Update: Ran some superficial tests:

Valve Index @ 120Hz 100% supersample (1.4x panel res)
After each scenario I change settings and reboot.

Radv alone (no amdvlk installed)
SteamVR Home
Floaty/Wobbly Dashboard: Yes
Repro-over/undershoot: Yes, when SS is increased
VRChat
Ron's Jump Hub FPS: 60
Loading screen Repro-under/overshoot: Yes
Floaty/Wobbly loading screen: Yes
SteamVR
Perf-Graph crashing/stuttering : yes
SteamVR needing an initial kill/restart to work: yes
"Extra empty options window": yes
```
 Notes: I chose Ron's Jump Hub because it's a pretty heavy world and reprojection is easy to trigger without people around. I do not test whether or not the "Binocular window" spawns on the desktop because that seems like a crapshoot. Above is what I normally run with.
```
Radv with amdvlk installed (no launch options set)
SteamVR Home
Floaty/Wobbly Dashboard: Yes
Repro-over/undershoot: Yes, when SS is increased
VRChat
Ron's Jump Hub FPS: 60
Loading screen Repro-under/overshoot: Yes
Floaty/Wobbly loading screen: Yes
SteamVR
Perf-Graph crashing/stuttering : yes
SteamVR needing an initial kill/restart to work: yes
"Extra empty options window": yes
```
 Notes: This was exactly the same as running with only radv.
```
Radv with amdvlk installed (SteamVR launched with "AMD_VULKAN_ICD=RADV %command%", VRChat launched with "AMD_VULKAN_ICD=amdvlk %command%")
SteamVR Home
Floaty/Wobbly Dashboard: Yes
Repro-over/undershoot: Yes, when SS is increased
VRChat
Ron's Jump Hub FPS: 60
Loading screen Repro-under/overshoot: Yes
Floaty/Wobbly loading screen: Yes
SteamVR
Perf-Graph crashing/stuttering : yes
SteamVR needing an initial kill/restart to work: yes
"Extra empty options window": yes
```
 Notes: Didn't make a lick of difference.
```

4 Radv with amdvlk installed (Using the method outlined in https://steamcommunity.com/app/250820/discussions/5/4839692156566904474/?ctp=3#c6163789298348219404)
SteamVR Home
Floaty/Wobbly Dashboard: Yes
Repro-over/undershoot: Yes, when SS is increased
VRChat
Didn't start
SteamVR
Perf-Graph crashing/stuttering : yes
SteamVR needing an initial kill/restart to work: yes
"Extra empty options window": yes

    Notes: Didn't make a lick of difference.

Radv with amdvlk installed (Using the method outlined in https://steamcommunity.com/app/250820/discussions/5/4839692156566904474/?ctp=3#c6163789298348219404, but switching around amdvlk and radv icd-files)
SteamVR Home
Only gives binocular window, regardless of killing/restarting steamvr
VRChat
Didn't get there
SteamVR
Didn't get there
```
 Notes: SteamVR (the app) really doesn't like amdvlk
```

I then removed the entry from /etc/environments and VRchat launch-options, rebooted, and started SteamVR. Now, even though it should be identical to scenario 2., the window-wobble was gone and there was no reprojection when turning up SS, and the perf-graph works. I got the "green artifacts around the lens-periphery"-bug as a tradeoff though.
I quit out of SteamVR and relaunched it without changing settings to doublecheck: back to wobbly windows and reprojection again.
Another thing I noticed: When the wobbly windows were gone and I had the green artifacts, performance was worse. I noticed my homeworld in VRChat got around 60 fps just spawning in and not even standing in front of the mirror almost like it was using legacy reprojection. When the wobbly windows were in effect the fps increased to around 120 fps at the entrance and hovered around 90 in front of the mirror. And I could swear I saw 120 fps in front of the mirror with scenario 4, so I recreated scenario 4... and now VRChat wont start with the amdvlk-launch options, only radv...

So... that's a wasted evening :-/

Answer 19 · 2023-07-27T09:28:50.000Z

I wish someone at Valve would spend some time on SteamVR for Linux, if the rumors are true the new standalone headset will need SteamVR running on Linux... This is the only thing keeping me using Windows right now, I was using Linux as my daily driver for years before I bought an Index and realized the claims of Linux support were abandoned

Answer 20 · 2024-01-21T16:22:15.000Z

I'm also affected.
I tried lots of things so far, including the VLK driver things from the guide, older versions of steamvr, turning off asynchronous reprojection, X11 gnome, kde Wayland.
Nothing worked.

Only happens under a good amount of load.
In SteamVR without a game, no issue. In VRChats Fractal Explorer from 1001, no issue. In VRChat worlds that are actually hard to run, instant wobble and jittering when looking around. It's actually so bad that if I would play like that for a few minutes, I would throw up for sure.

Currently I can't use VR at all, since Windows has another issue for me where one of the Vive trackers 2.0 will not stay connected if its below belly height (where I use it for full body tracking).

So no VR in Linux because I will throw up, no VR in Windows because I can't actually use my full body.

Valve please fix this issue... I'm happy to help however I can, literally just do something.

Answer 21 · 2024-01-21T20:17:02.000Z

Currently I can't use VR at all, since Windows has another issue for me where one of the Vive trackers 2.0 will not stay connected if its below belly height (where I use it for full body tracking).

This sounds like tracking occlusion, something is blocking the direct line of sight to your base station(s), such as clothes, bed, desk, etc.

Answer 22 · 2024-01-21T21:40:39.000Z

Currently I can't use VR at all, since Windows has another issue for me where one of the Vive trackers 2.0 will not stay connected if its below belly height (where I use it for full body tracking).

This sounds like tracking occlusion, something is blocking the direct line of sight to your base station(s), such as clothes, bed, desk, etc.

I appreciate the comment but its not the case. Same room, same PC, same trackers, same basestations. Using windows the Tracker wont work, using Linux it works perfectly. The only difference is the OS.

And its cut and dry as I make it seem. In Windows I can have the thing be direct line of sight, no obstacles, no mirrors or shiny surfaces, and still it wont work if its under my belly height. Meanwhile, using Linux, same location, same room same everything, it works flawlessly.