ARCore tracking is jumping despite 30 fps

Question

ARCore tracking is jumping despite 30 fps

ChristophGeske opened this issue 6 years ago · 6 comments

Using lerp to smooth out the jumps is possible but than every quick movement of the user also gets smoothed. This solution is not ideal therefore I have to work on a fix.

One possible solution seems to use the accelerometer to check if the data coming from the arcore camera are correct.

The idea of using the IMU was mentioned here
google-ar/arcore-android-sdk#604
by one of the google arcore team members.

Another fix might be to use a filter algorithm which cuts off movements which are to fast for humans or which are unlikely. For example one could record the last few frames and if the next frames differs to much from the ones before it could be a jump in the tracking and filtering it out might help to eliminate the jumps keeping responsiveness high and lag low.

Answer 1 · 2019-02-17T12:32:27.000Z

ARCore works at 10fps internally, being fused with IMU to interpolate to 30fps.

Answer 2 · 2019-02-17T13:20:24.000Z

Yes 30 fps is not the real camera frame rate, but the ARCore team claims they can track with 30fps by using sensor fusion but in reality I can not trust the 30 fps I receive from ARCore. I think I found a solution to that issue which I will publish soon. I use a filter to remove these jumps/tracking errors. This is an interesting problem where lots of brain power could be invested in to improve it further in the future.

Answer 3 · 2019-02-18T12:55:27.000Z

I wonder if it will support Snapdragon 845 / 855 Spectra ISP XR1 capabilities, that can handle 60fps positional tracking

Answer 4 · 2019-02-18T14:30:02.000Z

That's an interesting question which I do not have an answer for and I don't know if google is interested in using the new capabilities of these chips in their ARCore tracking solution. The problem for us who wanna use ARCore for VR is that we really need 60 fps but for the AR experiences google is using 30 fps is good enough. It might require google to change the software I don't know if just switching to a chip with one of the mentioned chips leads to an instant improvement.

I still have the hope that the imprecise data delivered by ARCore is good enough an can be used to simulate a 60fps head tracking with no noticeable artefacts. That would be the best case and massively increase the user base.
I think a filter function which removes errors in the ARCore tracking and which I am working on right now could be the solution and I can confirm that it improves the tracking noticeably. My current solution still has some artefacts but I am close to fixing all the bugs and will publish the first working version of the filter very soon.

Answer 5 · 2019-02-18T17:39:54.000Z

have you tried iPhone ARKit too ? I seems the solution you are developing is very interesting, I hope it works, good luck. Att Renato Wisocki Junior renato@wisockijunior.com.br Em seg, 18 de fev de 2019 às 11:30, Christoph Geske < notifications@github.com> escreveu:

…

That's an interesting question which I do not have an answer for and I don't know if google is interested in using the new capabilities of these chips in their ARCore tracking solution. The problem for us who wanna use ARCore for VR is that we really need 60 fps but for the AR experiences google is using 30 fps is good enough. It might require google to change the software I don't know if just switching to a chip with one of the mentioned chips leads to an instant improvement. I still have the hope that the imprecise data delivered by ARCore is good enough an can be used to simulate a 60fps head tracking with no noticeable artefacts. That would be the best case and massively increase the user base. I think a filter function which removes errors in the ARCore tracking and which I am working on right now could be the solution and I can confirm that it improves the tracking noticeably. My current solution still has some artefacts but I am close to fixing all the bugs and will publish the first working version of the filter very soon. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEixgqNCE1tME2FhLm9TWAuN5fKovNiUks5vOrjrgaJpZM4ZJq0b> .

Answer 6 · 2019-02-18T20:05:41.000Z

No I haven't worked with ARKit but even if it would deliver slightly better tracking compared to ARCore it would only work in combination with the Cardboard VR SDK which is significantly worse compared to the Daydream and GearVR SDKs. Daydream and GearVR use some special hard and software solutions making the experience less nauseating by reducing lag.