Vuforia only render in OpenGLES2, not Metal
igrir opened this issue · 13 comments
Hello, I'm having an issue after integrating react-native with Vuforia. Everything runs well, but Vuforia always runs on OpenGLES2 instead of Metal. I really need to run it on Metal because in the app we also need to run the ARKit. I tried to force Graphics API on Unity by change it only to Metal but the result was the same!
Here's the log when I am not using react-native-unity-view:
....
Initializing Vuforia...
2019-08-06 13:44:58.044546+0700 unityreact4[975:63447] User pre-granted access to the camera
2019-08-06 13:44:58.044615+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:58: Vuforia selected rendering API Metal
2019-08-06 13:44:58.044632+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:58: Vuforia SDK version 8.3.8
2019-08-06 13:44:58.044646+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:58: Vuforia SDK build 3621
2019-08-06 13:44:58.097371+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:58: Matched precache profile for Apple, iPad7,5, iOS 12, 800.1
2019-08-06 13:44:59.658717+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:59: Vuforia Fusion: Detected and enabled use of ARKit
2019-08-06 13:44:59.660540+0700 unityreact4[975:63447] Rendering mode set to: Metal
2019-08-06 13:44:59.672348+0700 unityreact4[975:63447] Rendering mode set to: Metal
2019-08-06 13:44:59.675556+0700 unityreact4[975:63447] DEBUG/AR(975) UIView has CAMetalLayer layer class
2019-08-06 13:44:59.675786+0700 unityreact4[975:63447] DEBUG/AR(975) UIView does not respond to selector renderFrameVuforia
....
And here's how it's reported after I used react-native-unity-view:
....
Initializing Vuforia...
2019-08-06 14:42:47.621329+0700 unityreact4[1053:77026] User pre-granted access to the camera
2019-08-06 14:42:47.621433+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:47: Vuforia selected rendering API OpenGL ES 2.x
2019-08-06 14:42:47.621476+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:47: Vuforia SDK version 8.3.8
2019-08-06 14:42:47.621504+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:47: Vuforia SDK build 3621
2019-08-06 14:42:47.690246+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:47: Matched precache profile for Apple, iPad7,5, iOS 12, 800.1
2019-08-06 14:42:48.997315+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:48: Vuforia Fusion: Detected and enabled use of ARKit
2019-08-06 14:42:48.999518+0700 unityreact4[1053:77026] Rendering mode set to: OpenGL ES 2.0
2019-08-06 14:42:49.036414+0700 unityreact4[1053:77026] Rendering mode set to: OpenGL ES 2.0
2019-08-06 14:42:49.037062+0700 unityreact4[1053:77026] DEBUG/AR(1053) UIView has CAMetalLayer layer class
2019-08-06 14:42:49.037287+0700 unityreact4[1053:77026] DEBUG/AR(1053) UIView does not respond to selector renderFrameVuforia
....
as you can see, it stated as Vuforia selected rendering API OpenGL ES 2.x. It's really strange. Maybe Vuforia detected the react app as OpenGL ES 2? Do I have to set something on XCode to force everything to render on Metal?
Thanks
@igrir As far as I'm aware, this project isn't fully compatible with Metal and is only set up for OpenGL ES 2 as per the docs:
IOS Platform:
Other Settings find the Rendering part, uncheck the Auto Graphics API and select only OpenGLES2.
@zbagley that's correct, but the thing @f111fei being aware of is the problem mentioned in this issue #29
I successfully included ARKit camera and run it using this library, but unfortunately I can't combine it with Vuforia as it only render black screen. The funny thing is the detection runs well! The 3D object being shown on the position where the marker was, but only the camera won't render
This could very well be a similar error that I've been running into. Ran a minimalist project from scratch (it uses ARFoundation ARKit) and ran into this for the first time when exporting:
@zbagley did you only add OpenGLES2 on your Graphics API? Might be that could be the problem. I've just tried it and also encountered that error.
For my issue, I don't know how Vuforia could select OpenGLES2 as I specifically only allow Metal for rendering from Unity Really weird 😅
The only Graphics API was OpenGLES2, when I add Metal it gives me il2cpp namespace errors:
If I only use OpenGLES2, the ARKit can't compile. From what I'm seeing the same trouble exists along EasyAR (from issue #123). Unless I'm missing something, it doesn't look like il2cpp supports ARKit/Metal. If that's the case, this project doesn't support ARKit.
I'll likely do a bit more exporation, but I'm already 40 hours deep into this and will likely try 2019.3 alpha with Unity as a library (#122) to see if it's more stable that this project and see what it would take to make the port.
@zbagley have you added -DNET_4_0 on your xcconfig? I tried to use ARKit utilizing Unity's AR Foundation and the camera seems working.
Unity as a library seems promising, and also being taken care of as it's official from Unity, it's superb. Maybe I'll do a little bit of research about it as well, but I don't know native development much. Hope we can integrate them using react-native as well later.
update with my research. By the way I also post the questions on Vuforia forum, feel free to join in discussion (https://developer.vuforia.com/forum/unity/vuforia-always-set-opengl-es-20)
I was reading how VuforiaUnityPlayer.h works, and looks like there's a method called initQCARiOS. I tried to find all of the scripts containg 'initQCARiOS' and suspect there's a part called Vuforia.UnityExtensions1.cpp which had this line:
int32_t L_4 = IOSUnityPlayer_initQCARiOS_m84BC6F102A88721658DDAF7926D5F52922E4972D(L_1, L_2, L_3, /hidden argument/NULL);
I noticed on top of that line there's a method called GetRendererAPI, which maybe called Unity's choice of render API. I change the output to 2 (Metal) as explained in here: https://library.vuforia.com/content/vuforia-library/en/reference/unity/classVuforia_1_1VuforiaRenderer.html
The result it was initialized in Metal. Here's the dump code I gathered:
Initializing Vuforia...
2019-08-07 10:20:08.453624+0700 unityreact4[1208:149363] User pre-granted access to the camera
2019-08-07 10:20:08.453676+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:08: Vuforia selected rendering API Metal
2019-08-07 10:20:08.453693+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:08: Vuforia SDK version 8.3.8
2019-08-07 10:20:08.453707+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:08: Vuforia SDK build 3621
2019-08-07 10:20:08.508179+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:08: Matched precache profile for Apple, iPad7,5, iOS 12, 800.1
2019-08-07 10:20:10.615716+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:10: Vuforia Fusion: Detected and enabled use of ARKit
but the problem persists.
I also tried the delayed initialization method because maybe Vuforia being initialized before Unity component being called (as explained from this article https://medium.com/@xinyustudio/vuforia-delayed-initialization-explained-3eddfd02ac6), but the screen still black (but of course, the image recognition still working).
I investigate further and found this part of dump log:
2019-08-07 12:10:24.144059+0700 unityreact4[1260:187356] INFO/AR(1260) 2019-08-07 12:10:24: Vuforia Fusion: Detected and enabled use of ARKit
2019-08-07 12:10:24.144959+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView has CAMetalLayer layer class
2019-08-07 12:10:24.148389+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView does not respond to selector renderFrameVuforia
2019-08-07 12:10:24.148626+0700 unityreact4[1260:187356] DEBUG/AR(1260) Could not find a UIView with CAEAGLLayer or CAMetalLayer layer class that responds to selector renderFrameVuforia
2019-08-07 12:10:24.148846+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView has CAMetalLayer layer class
2019-08-07 12:10:24.149036+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView does not respond to selector renderFrameVuforia
2019-08-07 12:10:24.149225+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView has CAMetalLayer layer class
2019-08-07 12:10:24.149422+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView does not respond to selector renderFrameVuforia
2019-08-07 12:10:24.149628+0700 unityreact4[1260:187356] DEBUG/AR(1260) Could not find a UIView with CAEAGLLayer or CAMetalLayer layer class that responds to selector renderFrameVuforia
2019-08-07 12:10:24.151296+0700 unityreact4[1260:187356] INFO/AR(1260) 2019-08-07 12:10:24: Vuforia resumed.
the interesting part was "DEBUG/AR(1260) UIView does not respond to selector renderFrameVuforia"
@igrir I must ask, did you run into any linking issues using RN 60.4? When linking I hit React/*.h missing type errors. When I create a custom podspec I run into UnityExport/*.h missing errors (added .podspec to node_modules library, imported react-native-unity-view in Podfile). Did you run into any of these sorts of issues, and how did you overcome them?
@zbagley Maybe this one? 👉 I did this which described by @ambewas on #122 (comment)
then in the library project itself, the header search paths need to be corrected, for React headers to be correctly resolved: add $(SRCROOT)/../../../ios/Pods/Headers/Public (or whatever relative path to that folder you've got) to library header search paths - set it to recursive.
Verified 100x now that all the libraries are ported in, all caches are cleared in XCode, etc. etc. Any standard Unity project works, but I still don't believe this project is compatible with AR (in any form) for iOS.
Using the 2019.1 ARFoundation sample https://github.com/Unity-Technologies/arfoundation-samples/tree/2.1 for the Unity project, and the simple ar scene, the exported simply doesn't resolve the AR architecture properly.
Giving the number of problems we've encountered we will likely not continue to explore this repo's compatibility.
Hi @igrir. I'm experiencing the same issue you described. Have you been able to find a solution or workaround?
Hi Good day. I see that this issue is still open. Is there any update on this?. I went through the same problems as you guys and our app works, but sometimes still crashes on startup, and I'm sure it is because of this. The app crashes when trying to execute UnityAppcontroller.mm - line 130 UnityInitApplicationGraphics(). Our app uses ARFoundations 1.5.0 preview 7, ARKit plugin 2.2.0 preview 6 in unity 2018.4.24.