This is something of a cold case, but I thought I'd take a crack at it. The problem is detailed here: https://gamedev.stackexchange.com/questions/85894/unwanted-darkening-at-polygon-edges-when-using-normal-maps-with-ssao
My answer to the original question "Is there anything obvious wrong with the normals?" is that there does appear to be something wrong, although it's not particularly obvious. When I finally saw it, I felt like Mac:
To begin, I wanted to emulate the normal colouring shown in the screenshots. At first I thought it matched the standard Unity normal colouring, but it doesn't. One difference, shown by the cube at the back right of the screen is that blue represents normals pointing towards the camera. In Unity, this is represented as negative in the blue channel and so clipped to 0, that is, black. The other difference, shown by the colouring of the floor, is that the normals are in world space. If it were in camera space, the floor would be apparently tipped towards the camera and mix some blue with the green.
This project contains a modified post processing debug layer that displays the normals in world space with blue towards us for comparison purposes.
Included in this project is the scene NormalTest
. This scene shows normal directions by reading them from the pixel under the mouse pointer and then projecting a line in the direction of the normal. This helps visualize the surfaces.
Focussing on the seat of the chair, broadly speaking it should be a flat surface pointing up and therefore should be a flat green colour. However, the normals have been smoothed. If I make a squashed cube and smooth the normals (the SmoothedNormals
scene), this is the colouring I get:
A smoothed and flattened cube: it is full green in the middle, tinting red to the right and blue to the front.
The seat of the original chair without normal mapping is roughly the same. It is full green in the middle, tinting red to the right and blue to the front.
But with normal mapping, in the middle it already appears tinted towards the blue, and then it tints further from there. This doesn't appear to be correct.
For comparison, here is an asset store chair: https://assetstore.unity.com/packages/3d/props/furniture/wooden-chairs-2-variations-125757. Without normals it shows a similar pattern.
With the normal map applied, you can see changes to colours in the detail, but on average the colours remain the same over the surface.
Focussing now on the legs, again broadly speaking, they should be cylinders coloured blue (because they should be facing you) and tinted red to the right.
Again, this is roughly what we see in the chair without normal mapping.
With normal mapping applied, it appears as if the normals have been twisted to the right.
Finally, looking at the back rest, this appears to be curved and so on the left hand side, I expect the normals to be rotated towards the right.
This is what we see in the unmapped version above.
However, in the normal mapped version, this tilt appears to have gone.
This suggests to me that either the normal maps have been generated incorrectly, encoded incorrectly, decoded incorrectly, or applied incorrectly.
To eliminate the possibility of a shader error I have taken the code provided and patched it into the Unity post process SSAO effect. It appears to work correctly without demonstrating the issues noted. In this project I have included a chair and an alien model from the asset store as test files. Both of these appear to render a decent looking SSAO effect, with and without normal maps. You can use the SSAO
scene to test this.
https://assetstore.unity.com/packages/3d/characters/creatures/alien-character-20838
There was some discussion in the original post about using a function to reconstruct the normals from the 3D position of each pixel. I have included an additional option to apply this function. In this case, the normal map is not used. I managed to get reasonable results from this function, but there was a bug in the code. I needed to swap the parameters of the cross product or I got bad results like those shown in the original post.
float3 reconstructNormal(float3 positionWorldSpace)
{
return normalize(cross(ddy(positionWorldSpace), ddx(positionWorldSpace)));
}
I suspect the normal maps are the cause of the original issue. The maps should not be making such large scale modifications to the normals. I would recommend checking known good normal maps and models with the engine to see if the issue was in the creation or application of the normal maps.
Using reconstructed normals, the cross product needs reversing to give a good result.