First let me apologize for asking what I am sure is an overly asked question but all my searching and testing has resulted in nothing.

I am attempting to remove all the GLSL built-ins from my shaders and the last one for #version 150 is gl_NormalMatrix. From all my reading it’s the transposed inverse of the upper 3x3 of the product of the Model and View matrices. So I have tried numerous permutations of this line (mat3 glNormalMatrix = transpose(inverse(mat3(mView * mModel)))) with no success replicating gl_NormalMatrix.

Below is a normal mapping vertex shader and it works perfectly with the gl_NormalMatrix built in but this built in is depreciated in version > 120 thus the exercise.

I am making some assumptions here but I am pretty certain may matrices are correct and I pass a projection, view(camera) and model(world) matrix in to transform the object coordinates. Additionally I have annotated what space I believe the vec3 to be in and what direction the mat4 is intended to transform to. I don’t think it’s a problem with my TBN but not ruling that out also. I just assumed I’d be able to calculate a ‘replacement’ for gl_NormalMatrix from the components passed in but that is proving to be harder than I had suspected. The offending line is in the CalculateTBN near the bottom with the working line uncommented and the ‘what I think is correct’ commented.

I don’t rule out the TBN calculations but I believe they are correct as they work with the gl_NormalMatrix.

If seeing the fragment shader or my TBN calculations will help I will post but I suspect those components may be a side matter to this activity. Also this shader is not optimized I know but I’m following the mantra, make it work (built-ins), make it work right (remove built-ins), make it fast (move some or most of the matrix math back to CPU). Maybe that exercise will solve this problem but I suspect I should be able to replicate a replacement for gl_NormalMatrix in the shader also. Additionally any insight into making it better is always appreciated.

Notes on what’s happening when rendering a plant with normal maps. With gl_NormalMatrix the lighting seems correct and stable to light source. Normal map looks correct, specular stay with camera angle. With my attempts to replicate the gl_NormalMatrix the dark side rotates with the globe, specular does odd stuff like going opposite directions, culminating at points on the globe and other odd effect. The normal mapping seems to be correct as it still looks accurate but with the shaded and specular moving it can be hard to tell.

Any insight is greatly appreciated…

Code :
//#version 150
 
uniform mat4 mProjection;                                                         // to clip space  (projection)
uniform mat4 mView;                                                               // to view space  (camera)
uniform mat4 mModel;                                                              // to world space (world)
 
uniform vec3 vLightSourcePosition;                                                // world space
uniform vec3 vCameraSourcePosition;                                               // world space
 
attribute vec3 a_vVertex;                                                         // incoming vertex (object space)
attribute vec3 a_vNormal;                                                         // incoming normal (object space)
attribute vec2 a_vTexel;                                                          // incoming normal (object space)
attribute vec3 a_vTangent;                                                        // to tangent space
/* ------------------------------------------------------------------------------------------ */
varying vec2 vTexCoord;
 
varying vec3 vLightDirection;                                                     // object space
varying vec3 vCameraDirection;                                                    // object space
 
mat3 CalculateTBN(vec3 vNormal, vec3 a_vTangent);
 
void main(void)
{
  vec3 vVertex = a_vVertex;                                                       // object space
  vec3 vNormal = a_vNormal;                                                       // object space
  vTexCoord = a_vTexel;
 
  // Calculate the positions from the model using the inverse of the model 
  //  Move from model space to local (object space)
  vec3 vLightPosition = vec3(inverse(mModel) * vec4(vLightSourcePosition, 1));    // object space
  vec3 vCameraPosition = vec3(inverse(mModel) * vec4(vCameraSourcePosition, 1));  // object space
 
  mat3 TBN = CalculateTBN(vNormal, a_vTangent);
  vLightDirection = TBN * (vLightPosition - vVertex);                             // tangent space
  vCameraDirection = TBN * (vCameraPosition - vVertex);                           // tangent space
 
  // Calculate the vertex position by combining the model/view(camera)/projection matrices
  gl_Position = mProjection * mView * mModel * vec4(a_vVertex, 1);                // clip space
}
 
mat3 CalculateTBN(vec3 vNormal, vec3 a_vTangent)
{
  mat3 glNormalMatrix = gl_NormalMatrix;                                          // to normal space??
  //mat3 glNormalMatrix = transpose(inverse(mat3(mView * mModel)));               // to normal space??
 
  // Calculate the Inverse TBN  (to transform from object space to tangent space)
  vec3 n = normalize(glNormalMatrix * vNormal);
  vec3 t = normalize(glNormalMatrix * a_vTangent);
  vec3 b = cross(n, t);
  mat3 TBN = transpose(mat3(t,b,n));                                              // to tangent space
 
  return TBN;
}