I wrote a shader a year or two ago that used normal maps. Off the top of my head I don’t remember exactly how it worked for certain. But I think the normals that go into the map are deviations from the flat triangle surface. So, if the normal vector in the map is default the pixel will point perpendicular with the flat face of the triangle. This means that the normals in the normal map are not related to the direction the model is facing, the side of the model the triangle is on, or the world origin. They are only related to the forward/tangent/perpendicular direction of the triangle currently being shaded. A blank normal map that has nothing but that familiar basic blue color will result in flat shaded triangles on your mesh even though all the triangles on the mesh face in different directions. The color that represents the pixel’s vector normal is a 3D vector that represents the deviation from perfectly perpendicular with the triangle’s flat surface.
Calculating a triangle’s normal is very straight forward if you know vector math. It’s the vector cross product. Done. To elaborate a little more, you can pick any of the 3 vertices in a given triangle and form vectors using the other two vertices that share a common origin (the first vertex). These vectors will perfectly represent the edges of the triangle in 3D space. And the cross product between those two vectors will give you a vector that points straight out of the triangle (is tangent to the triangle). You may need to normalize some of these vectors. I haven’t done the math in awhile and often times you will get the wrong results if vectors are not normalized. Plus, the output should be a normal as well. You may be able to get away with only normalizing the result. (I always forget when you need to normalize. I think it’s actually the dot product that I’m thinking of. I always have to look it up, but these formulas and algorithms are all over the Internet. And I normalize for good measure if I’m unsure any time the length/amount of the vector is irrelevant. Technically it’s not a normal unless it has a length of 1. And with some calculations you can end up de-normalizing normals.) Check out my video on vectors around 1 hour 39 minutes where I talk about vector cross products and multiplication.
So, once you have the triangle’s normal, you can rotate is by the normal in the normal map.
This webpage shows the process. They call the triangle’s normal that I just described how to calculate as the “Up vector”. This could get confusing here because we have tangents of tangents. The tangent and bi-tangent they describe are on the surface of the triangle, not pointing out from it. They seem to be determined as Tangent being the horizontal UV coordinate direction and the Bi-Tangent being the vertical UV coordinate direction on the face of the given triangle.
The three vectors together are mutually perpendicular and form a “private axis (or origin)”. This is what goes into a 3by3 rotation matrix. So, you’ve basically build a 3by3 rotation matrix here that describes the orientation of the triangle face in 3D space.
I’m pretty sure there’s a way to calculate the triangle’s normal in the fragment shader. In HLSL there are some functions that let you tap into the pixels next to the current pixel and their positions or in 3D space. So, you can calculate the tangent to the triangle using these three 3D positions with the vector cross product as previously described. I’ve done that before as well, just not with normal mapping.
I might also point out what the normal map colors are. When you build the normal map, you get a vector that is perpendicular to the pixel. It’s the direction the pixel is facing and it maps to an exact pixel/texel in the UV map. 3D vectors can point in all directions including negative ones. This is a normal; so by definition it has a length of 1 unit. But considering the negative direction it can be anywhere between -1 and +1 in length along an axis. Color space does not have negatives. There is no such thing as negative red, green, or blue. So, if you shift this normal by +1 you can map it into the range of 0 to 2. But the maximum color value is 1. So, you divide it in half giving a “normal” value between 0 and 1. This can be assigned an rgb color where red is x, green is y, and blue is z for example. Then you can store every pixel’s normal in an image and save it as a .JPG or .PNG or whatever. To load it you reverse the process. Multiply by 2 and subtract 1.
But just remember: any time you have two vectors that share a common origin in 3D space (like two edges on a triangle or rectangle that meet at a corner), their cross product yields a vector that points straight out of the 3D plane they both live on. I like to think of it as converting 2D space into 3D space because it’s the direction to go to get out of the 2D plane. Reversing the order of multiplication in the cross product will give you the vector on the other side of the plane that points out of the plane in that direction. Any time you see a vector cross product, this is likely what it’s doing.