Skinned, morphed, normal-mapped models have incorrect lighting in SceneKit
lenkawell opened this issue · 15 comments
The attached Velociraptor.glb model file (converted from FBX via Sketchfab) does not display the Body_Mat correctly in SceneKit with either GLTF Viewer or our app. The model material is displayed as a mottled gray via SceneKit but it displays correctly in babylonjs.com, gltf-viewer.donmccurdy.com and Gestaltor. Sorry, but I haven't investigated it in any detail to determine what might be the problem.
Something about the normal map is going very wrong here, but I haven't found the root cause yet.
The tangent frame values are insane in the shader. Probably not an issue with the normal map per se, but instead something to do with the normal/tangent data.
The tangent geometry source is full of zeros, leading to the incoming tangent and bitangent in the fragment shader to be NaNs...but only when morph targets are enabled. There may be a SceneKit bug or regression here, but more urgently, our whole approach to filling in missing morph target attributes is bogus because of the use of the additive morph mode. I'll rewrite that logic then try to debug this.
Thanks for looking into this so quickly!
This is at least partly a regression in macOS Sonoma (and I presume iOS 17). The tangents produced by the morphing shader are all 0, leading to the tangent frame used for normal mapping being completely bogus. I've filed this as feedback with Apple (FB13340109).
I've also created a branch to explore possible improvements in our handling of skinning and morphing, but it's hard to know whether the changes there will improve anything until Apple fixes this blocker. In the meantime, perhaps the least invasive change is to disable normal mapping when morph targets are present.
This is a known issue being discussed over on the Apple developer forums. Users there seem to have largely come to the same conclusion that I came to independently. Purportedly it was at least partly fixed in iOS 17, but very obviously there's still something wrong (at least on the latest public build of macOS).
The SceneKit bug indeed seems to be the problem. Using our app I verified that, after loading the model and removing either the normal map or the morpher will fix the lighting problem, of course with a loss of either fidelity or functionality. I tested it and it's still broken on both macOS 14.2 Beta (23C5030f) and iPadOS 17.2 Beta (21C5029g).
Appreciate the further testing. I'd recommend filing your own feedback with Apple so this gets more visibility. Although it probably won't make the macOS 14.2 GM, it's shallow enough and severe enough that we might see it fixed before the next major versions.
This seems to have been fixed in macOS Sonoma 14.3 Beta (23D5033f). I'll provide additional updates after additional testing.
I went ahead and merged my changes to morph target and skinning attributes as 0dec7e7. Current plan is to cut a release once it's had time to soak and I can test it on iOS 17.3 beta.
Verified on i(Pad)OS 17.3 beta (21D5026f).
Morph target improvements have been incorporated into release 0.5.5. Full resolution of this issue is pending on the GM release of macOS Sonoma 14.3 and iOS 17.3.
Now that our fixes have been incorporated and the relevant OS releases are widely available, I don't think there's anything left to do here. This issue will be closed in seven days if no further information or difficulties come to light.
Closing.