"I’m trying to transform a point by the inverse of the modelview matrix. Can someone tell me if this is correct :
v’ = Mv where M is the inverse modelview"
Obviously, if v’ is your point and v the transformed point, this is correct. If v is your point and v’ your transformed point, then you forget to inverse M
(I must confess I didn’t understood a bit of your problem)
Originally posted by moko: The point is a 2D coord, so to make it a 1x4 column vector it becomes (x,y,0,1)?).
That will work as long as you have not translated along Z (or you are using ortho projection), or rotated around X or Y, e.i. the third row of M is (0,0,1,0).
Originally posted by moko: In the resulting vector, I will have to divide the 1st 3 elements by the 4th, to make it equal to 1(for homogenous coord).
As long as you haven’t done anything weird, the W value will be 1, because the 4th row of M will be (0,0,0,1). So there is no need to divide.