How do I rotate a vertex by the current modelview matrix?

OpenGL OpenGL: Basic Coding
1

Hi list,

I’m having some problems doing depth sorting, because I’m stuck on one thing. I
can extract the x,y,z translation from the modelview matrix, but just using
that straight with a 3d pythag to find the distances of my billboards leads to
draw errors. By examining the modelview matrix I realised that I need to rotate
the x,y,z translation coords that I extract from the model view matrix.

Only problem, I’ve never done vector rotation, and also never in combination
with OpenGL.

I know how to extract the rotation matrix. Or even if I don’t know I can work
it out by doing glrotatef and then examing the modelview matrix to see what has
changed.

But the problem is, what do I do with the x,y,z rotations? Does OpenGL suffer
from gimbal lock using Euler angles? Or do I do some full matrix rotation? Like
I said, I’ve never done vector rotation, and I don’t know what type of vector
rotation OpenGL uses (I know there are several: Euler Angles, quaternion,
matrix, and more)

2

You can extract the matrix as usual, into a 16-element array. Then you multiply the homogenous point in worldspace (x, y, z, 1) with this matrix, and you have the point’s coordinates relative the origin. Then you calculate the distance from the origin to the transformed point using pythagoras theorem. Then you have the distance from the viewpoint which you can use for depthsorting.

But remember, depthsorting is not always needed. You don’t say what kind of things you are drawing. If it’s a particlesystem with light emitting particles, you can get away with an additive blending function (glBlendFunc(GL_ONE, GL_ONE)), and then there is no need to depthsort, as long as you draw the particles after any other opaque object(s). But then remember to turn depthtest off.