What is the right way to invert transformation matrices? I need for back face culling.
And what part of back face culling does glEnable(GL_CULL_FACE) take in account? And clipping?
If the transformation matrix is just a rotation matrix then the inverse is just simply the transpose of the matrix. Otherwise it is more involved. From “Matrices and Transformations” (A. J. Pettofrezzo), “in general the inverse of a square matrix is the product of the reciprocal of the determinant of that matrix and the transpose of the matrix of cofactors.”
[This message has been edited by DFrey (edited 07-25-2000).]
Inverting the standard transformations like rotation, translation and scale is also possible with an own matrix stack which exactly does the inverse of the applied transformations in reverse order.
OpenGL matrices are left multiplied, so something like
v’ = M * v
Let’s say you have transformations
glTranslate()
glRotate()
glScale()
in you code, then the calculation looks like
v’ = T * R * S * v
If you want the inversion of (TRS) you need
(TRS)^-1 which is S^-1 * R^-1 * T^-1
because (mind the vector on the left is the original !)
v = S^-1 * R^-1 * T^-1 * T * R * S * v = I * v
where each matrix
(T^-1 * T) = I
(R^-1 * R) = I
(S^-1 * S) = I
is the identity
So for each matrix operation you could do the same in opposite direction (for translation or rotation) or 1/factor (for scaling) and always have an inverse matrix.
Long story, for the straightforward approach there will defintely be some OpenGL matrix inversion routine in the MESA implementation. 
>>And what part of back face culling does glEnable(GL_CULL_FACE) take in account?<<
Face culling is done based in the winding order of the vertices projected onto the projection plane. Means, if your winding direction for front faces is counterclockwise glFrontFace(GL_CCW) is default, then a triangle which comes out on the display with the same order of vertices will be displayed, the ones viewed from the back have clockwise order and won’t be displayed. That’s all.
This is also the reason why normals for lighting purposes have always to be specified for the front face direction
>>And clipping?<<
Which one, view frustum or user clipping?
OpenGL always clips any primitive against the 3D view frustum. Nothing there to enable or disable.
[This message has been edited by Relic (edited 07-26-2000).]
To expand on DFrey’s answer… rotation matrices are the easiest to invert (you don’t even need to build a separate inverse matrix, you can just multiply straight through the transpose) but not all other matrices require the full-blown three-pages-of-gibberish required by an arbitrary inverse. Rigid-body matrices (rotation and transform only) can be handled as a transpose and a vector add. The Red Book appendix has the formulae for calculating inverse projection matrices.
In practice, this probably covers most cases. I don’t have an arbitrary matrix inverse routine, and have never felt the need for one.
And yes, there is arbitrary-inverse code in Mesa, though I wouldn’t describe it as “straightforward”! 
To be more precise, I use an extra byte with each my matrices to keep track of the type of operations that have been used on it, arbitrary load, rotation, translation, scaling, identity load. Some operations clear bits, and others set one or more bits obviously. Then when I need the inverse I check those bits to determine the best way to find the inverse.
Interesting… I use C++ strong typing to do the same sort of thing (i.e. you declare a RotationMatrix, and the compiler makes sure it stays a RotationMatrix).
Upside is that you can also use the strong type guarantees to automatically optimize matrix * vector ops etc without any conditional overhead. Downsides are some loss of flexibility, (though in practice I haven’t found any problems yet in this respect) and the fact that methods taking matrix params would have to overload on all types they wanted to optimize for (I’m not using virtuals, for obvious reasons).
Have to think about this some more…
[This message has been edited by MikeC (edited 07-26-2000).]
>>And yes, there is arbitrary-inverse code in Mesa, though I wouldn’t describe it as “straightforward”! <<
Hi MikeC, I didn’t mean that the code in Mesa is straightforward, but the “I have a matrix and want to know the inverse” solution instead of the bunch of stuff I wrote before.
I have found it now and it’s not a secret: http://www.acm.org/tog/GraphicsGems/
It’s in the Graphics Gems I (gems.zip)
Look for Richard Carling Matrix Inversion.
Have fun.
[This message has been edited by Relic (edited 07-26-2000).]