Is it possible to use the inverse of the MODEL_VIEW matrix for undoing the rotation and translation of the camera frame?
I have tried but it doesn’t seem to be consistent - I have problems with the fact that the normal to the image plane always seems to be oriented in the positive z-direction after the modelview transformation and there is no way of telling where it was originally. This doesn’t follow from the definition of the model_view matrix though.
I tried coding this and for the following:
gluLookAt(0.0,0.0,115.0,0.0,0.0,0.0,0.0,1.0,0.0)
glFrustum(-60.0,60.0,-60.0,60.0,40,200)
I get the model_view matrix:
1 0 0 0
0 1 0 0
0 0 0 -115
0 0 0 1
However, after the projection, all the z-coordinates on the image plane are positive, as if the normal of the image plane was pointing in +z, but it is not (I positioned the camera on the +z axis and looking towards the centre, which means the normal is (0,0,-1), but the rotation part of the modelview matrix in my case is an identity?!? )
The frustum call builds a matrix that defines what the post projection space is. Below is the matrix from the man page, note that there is a -1 in the 4th row 3rd column that explains your z direction problem.
2 zNear
------------ 0 A 0
right - left
2 zNear
0 ------------ B 0
top - bottom
0 0 C D
0 0 -1 0
A = (right + left) / (right - left)
B = (top + bottom) / (top - bottom)
C = (zFar + zNear) / (zFar - zNear)
D = - (2 zFar zNear) / (zFar - zNear)
It also seems that you aren’t distinguishing between the modelview matrix and the projection matrix. Undoing the modelview is just a case of putting identity on the modelview, provided you have used the projection matrix for projection (and you really want to do that).
If you try to undo modelview after you’ve multiplied projection you could very possibly have lighting screwed up because lights are stored in eye space i.e. in the space between the modelview and projection matrices and that should be before projection. On top of this undoing the model transformation post projection is not the same as undoing it pre projection, you will get bogus results.