Shrink frustum to specific screen-space rectangle

[Ilian Dinev]

For frustum-culling purposes, I need to modify the left/right/top/bottom planes to encapsulate a smaller volume.

Existing structs are simply:

struct Frustum{
	vec4 planes[6];
 
	void ConstructFromMatrix(const mat4& ViewProjection);
};

The required function/method is:

void Frustum::ShrinkToRect(int x=77,int y=93, int width=128,int height=64,    int ScreenWidth=1280,int ScreenHeight=720);

Sanity check: if the func is called with the following args, there is no change in the frustum data:
ShrinkToRect(0,0,1280,720, 1280,720);

My model/view/projection spaces are all with +Z pointing forward (not back). (left hand coords) +Y points up, +X points to the right. The screenspace coords are +X to right, +Y to up.

I attempted approaching the problem in several ways, but failed - possibly some of my maths was buggy. One of the approaches was:
- project a quad on the near-plane, vertices at the specified x/y/x+width/y+height positions when projected by the ViewProjection matrix.
- another quad on the far-plane, same calcs.
- construct the 6 planes out of the 8 resulting vertices


All the matrices and their creation args are available (I'm not dealing with external code), and optimizations aren't necessary.

Any help is appreciated.

[Stanley L]

Have you tried using glFrustum (or the equivalent projection matrix listed in the documentation) to form your 6 planes? It should be able to generate the off-axis asymmetric projection that you need. I figure all you would need to do is calculate new top, bottom, left and rights using the appropriate percentages based on the ShrinkToRect parameters.

ShrinkToRect(640, 360, 640, 360, 1280, 720) should generate a quarter sized view frustum which, I think, would look something similar to glFrustum(0, origRight, 0, origTop, zNear, zFar) if I've got things right. (I've taken the liberty to assume that your original projection was symmetric in this example.)

[Ilian Dinev]

Thanks a lot! This did exactly what I wanted. Will need to be used together with glViewport() and optionally scissors, just as I hoped.