Eye position in object space?

[mattjakob]

I know it's very dumb but I cant get around this.
I need to calculate the position of the camera (eye) in object space, given a transformation matrix M.

I think in theory is should be:

eye_obj = col * (V.transpose).inverse * M.inverse


Where:
col = {0,0,0,1}
V = 4x4 camera transform matrix retrieved via glGetFloatv(GL_MODELVIEW_MATRIX, V)
M = 4x4 transform matrix defining the center of the object

[DmitryM]

I guess you need:
eye_obj = col * V * M.inverse = V.pos * M.inverse

[Dark Photon]

I need to calculate the position of the camera (eye) in object space, given a transformation matrix M.

I think in theory is should be:

eye_obj = col * (V.transpose).inverse * M.inverse
Dmitry's got you covered, but it looks like you've got it, except for that transpose in there.

MODELVIEW = Object-to-eye-space transform

so...:

v_eye = MODELVIEW * v_obj

But you want the opposite, so multiply both sides by INV_MODELVIEW:

INV_MODELVIEW * v_eye = INV_MODELVIEW * MODELVIEW * v_obj
INV_MODELVIEW * v_eye = v_obj
inverse(M) * inverse(V) * v_eye = v_obj

So now just plug-n-chug. The eye-space vertex position (v_eye) you've got is (0,0,0,1) -- the eyepoint. So plugging it in, this will effectively just extract out the translation component in INV_MODELVIEW.

You can multiply V*M and invert the result to get INV_MODELVIEW, or you can invert M and V separately, and multiply them in the opposite order to get INV_MODELVIEW. Then multiply that by the vector (v_eye) of (0,0,0,1).

Note that my text above uses OpenGL transform ordering (operator-on-the-left -- i.e. Ax=y). It looks like you're using the opposite in your notation (what you'd expect for C++), so just flip everything left-to-right. That said, looks like you already have the right answer, except for that transpose.