Maybe use GL_STENCIL_TEST ?

[pleopard]

I could use a little guidance on how to approach this effect ...

This is what I am doing ...

I have a missile traveling over my terrain and it is tracking a target on the ground. Its search pattern is a simple frustum defined by an azimuth FOV and an elevation FOV. So, in each frame, after rendering the missile and its target, I draw a transparent frustum from missile to just a little past the target. I can see the missile's search cone superimposed on the target and its boundaries all the way back up to the missile.

This is what I want to do ...

I want to be able to project the search pattern onto my terrain and not have to draw the frustum. That is, all of the terrain area within the search frustum's projected boundaries will be discolored or outlined somehow.

I could trace four rays from the missile to the four corners of the search pattern on the ground and then do a terrain height query at regular intervals between these points to get the projected outline but that would be extremely expensive computationally (like 0.1 seconds per frame or more for my situations).

Can't I achieve this effect somehow with the stencil buffer (GL_STENCIL_TEST)? Has anyone attempted a similar problem as this? If so, can you give me any advice/guidance or point me to a good resource that would educate me on such a technique?

[jwatte]

This is exactly like stencil volume shadows. Google for that and you'll get TONS of hits. In brief, here's how it works:

0) Create frame buffer with 8 bits of stencil.

1) Create a closed mesh that encompasses exactly the search area. For a pyramid, this is very simple.

2) Clear stencil to 0. Make sure the depth buffer is fully rendered. Color does not need to be rendered at this point.

3) Turn on face culling, and cull out BACK faces.

4) Render the volume mesh into frame buffer, using Z testing, but no Z or color write, updating the stencil buffer by adding 1 for the fragments that pass the test.

5) Change culling to cull out FRONT faces.

6) Render the volume mesh into frame buffer, using Z testing, but no Z or color write, updating the stencil buffer by subtracting 1 for the fragments that pass the test.

Any pixels that are inside the volume you rendered will pass the first test, but not pass the second, and thus you'll have a non-0 stencil value for those pixels.

Now, set stencil testing to only affect the frame buffer where the stencil is non-0, and render whatever you want. This could be a re-render of the terrain in a different color, or it could be a full-screen red semitransparent quad, or something else.

The is the "Z-pass" method which has problems when the camera is inside the volume mesh (i e, the mesh isn't actually closed WRT the rendering frustum). There are alternatives, such as "Z-fail" in combination with infinite far plane projection matrices, which may be more robust -- again, google for these buzz words, or check the NVIDIA developer site for their robust shadow volumes whitepaper.

[pleopard]

Wow, thanks ... It may take me a while but I will give that a shot. Re-rendering the terrain isnt an option but the fullscreen quad may do it for me.

[dorbie]

Here is something similar I did, it may be of some use.

http://www.sgi.com/software/performer/brew/uav.html

In addition to the stencil test it uses projective texture to apply an image to the terrain frustum. It includes occlusion calculations using a projective depth map shadow test.