I found the post. It wa son the opengl-gamedev-l list. Read and enjoy:
There is a far easier and faster way to get the frustum planes from the
projection matrix. The 4 plane coeficients for the six planes are obtained
almost directly from the projection matrix as:
row[3] + row[0]
row[3] - row[0]
row[3] + row[1]
row[3] - row[1]
row[3] + row[2]
row[3] - row[2]
(Where row is definited as in the opengl doc). The nice thing about this
method is it works with any projection matrix at all. Also, if you subtract
the rows of the view matrix times the projection matrix, you get the planes
in object space which is very handy for culling. Below is an old post that
tom wrote up about this stuff.
-Gil
So you want to get the current view frustum into an objects coordinate
system as quickly as possible? Well, here’s the code!
void ExtractPlanes(CPlane Planes[6])
{
C4Vector tmpVec;
float fTmpLength;
C4Matrix ProjMat, MViewMat, ComboMat;
glGetFloatv(GL_PROJECTION_MATRIX, ProjMat[0]);
glGetFloatv(GL_MODELVIEW_MATRIX, MViewMat[0]);
ComboMat = MViewMat * ProjMat;
ComboMat.transpose();
tmpVec = ComboMat[3] - ComboMat[0];
Planes[0].normal().set(-tmpVec[0], -tmpVec[1], -tmpVec[2]);
Planes[0].distance() = -tmpVec[3];
tmpVec = ComboMat[3] + ComboMat[0];
Planes[1].normal().set(-tmpVec[0], -tmpVec[1], -tmpVec[2]);
Planes[1].distance() = -tmpVec[3];
tmpVec = ComboMat[3] - ComboMat[1];
Planes[2].normal().set(-tmpVec[0], -tmpVec[1], -tmpVec[2]);
Planes[2].distance() = -tmpVec[3];
tmpVec = ComboMat[3] + ComboMat[1];
Planes[3].normal().set(-tmpVec[0], -tmpVec[1], -tmpVec[2]);
Planes[3].distance() = -tmpVec[3];
tmpVec = ComboMat[3] - ComboMat[2];
Planes[4].normal().set(-tmpVec[0], -tmpVec[1], -tmpVec[2]);
Planes[4].distance() = -tmpVec[3];
tmpVec = ComboMat[3] + ComboMat[2];
Planes[5].normal().set(-tmpVec[0], -tmpVec[1], -tmpVec[2]);
Planes[5].distance() = -tmpVec[3];
}
Of course you need to have a bit more info before this to be useful …
- C4Vector is a class with an array of 4 floats for data and a bunch of
operators for addition, subtraction, etc.
- C4Matrix is a class with an array of 4 C4Vectors for data and a bunch of
operators for addition, subtraction, etc.
- CPlane is a class with a C3Vector for the normal and a float for the
distance.
- I want the normals of the planes of my frustum to point outside the
frustum (if you want them to point inside then remove the negation of the
normal and distance)
- If you are so inclined you can normalize the planes by dividing the
normal and the distance by the length of the normal, but it isn’t necessary
for culling algorithms to work (verified … I’ve used it both ways). The
only reason to normalize is so you can “see” the normals better when
debugging.
- I’ve been told that this should work under D3D in DX7 as well (but not
previous versions). If someone can confirm/deny this I’d appreciate hearing
about it.
- Here is my matrix multiplier routine. If yours works differently then
you can reverse the order of the multiply to ProjMat * MViewMat.
const C4Matrix C4Matrix::operator*(const C4Matrix _rhs)
{
C4Matrix dst;
for ( int j = 0; j < 4; j++ )
{
dst[0][j] = (vec[0][0] * _rhs[0][j] +
vec[0][1] * _rhs[1][j] +
vec[0][2] * _rhs[2][j] +
vec[0][3] * _rhs[3][j]);
dst[1][j] = (vec[1][0] * _rhs[0][j] +
vec[1][1] * _rhs[1][j] +
vec[1][2] * _rhs[2][j] +
vec[1][3] * _rhs[3][j]);
dst[2][j] = (vec[2][0] * _rhs[0][j] +
vec[2][1] * _rhs[1][j] +
vec[2][2] * _rhs[2][j] +
vec[2][3] * _rhs[3][j]);
dst[3][j] = (vec[3][0] * _rhs[0][j] +
vec[3][1] * _rhs[1][j] +
vec[3][2] * _rhs[2][j] +
vec[3][3] * _rhs[3][j]);
}
return dst;
}
To go along with this I also have an AABB culling routine.
int IsVisible(CPlane Planes[6], CAABBox AABB)
{
C3Point MinPt, MaxPt;
bool bIntersecting = false;
for (int i = 0; i < 6; i++)
{
for (int j = 0; j < 3; j++)
{
if (Planes[i].normal()[j] >= 0.0f)
{
MinPt[j] = AABB.min[j];
MaxPt[j] = AABB.max[j];
}
else
{
MinPt[j] = AABB.max[j];
MaxPt[j] = AABB.min[j];
}
}
if (Planes[i].dist_to_point(MinPt) > 0.0f)
return 0;
if (Planes[i].dist_to_point(MaxPt) >= 0.0f)
bIntersecting = true;
}
return bIntersecting ? 1 : 2;
}
Return Values:
0 = Outside
1 = Intersecting
2 = Inside
It will report some cases as being intersecting when in fact they are
outside. How much of a concern this is depends entirely on how your data is
set up. An example of a false intersecting case can be seen here at: http://www.3dgamedev.com/projects/falsepos.gif.
The AABB test is right about of the Real Time Rendering book with the
following caveats(authors are CC’d on this e-mail):
- The book’s test doesn’t differentiate between inside and outside
- The book’s pseudo code (on page 311) has an error in it (the else’s
should be removed)
Credit goes out to Gil Gribb for coming up with the method for extracting
the planes and helping me get all the signs right and working correctly.
Tom