I have two meshes that intersect, like so:
I would like to remove the part of the ball that pokes above the trough, like this:
I guess this falls under the heading of “constructive solid geometry”. If I draw the top surface of the trough into the stencil buffer, I can remove the part of the ball that is in front of the top surface, but not all of the ball above the trough, like this:
I suppose it should be possible to do what I want using the stencil buffer, if I could think of a good way to put the appropriate mask in the stencil buffer. Suggestions?
Ok, since you allready have (partially) working solution with stencil buffer I’ll just suggest to add invisible polygons to your clipping surface that will cause it to become closed brush. Just draw these faces with glColorMask set to 4x GL_FALSE.
Interesting idea… OK, so I make it a closed volume by adding invisible surfaces, shown here as translucent for purpose of illustration…
If I set the stencil buffer by drawing the top surface of the trough and the front faces of the invisible surfaces, it looks like that would work in this case. But here’s another camera angle on the same situation. (I changed the relative sizes of the objects.)
We can see two areas of the ball through the top surface of the trough. One of them should be clipped out (the oval area), and one should not (the sliver at the top). It seems tricky to make that distinction.
Here’s a possibility that I thought of:
[ol][li]Draw the ball into the depth buffer.[]Turn off writing to the depth buffer, but leave depth testing on.[]Turn off backface culling.[]Set the stencil test to always pass.[]Set the stencil test actions (glStencilOp) so that if the stencil test passes and the depth test passes, the stencil bit is inverted, and otherwise the stencil bit is unchanged.[/ol][/li]
If I’m not mistaken, this should put 1s in the stencil buffer in the oval area where I want the ball clipped out, and 0 (because of double invert) in the sliver of the ball that should not be clipped. The problem with this idea is that I’m actually drawing these two objects as part of a larger scene, and I’d rather not mess up the depth buffer. I guess I could save and restore the depth buffer with glReadPixels and glWritePixels, but that would probably be bad for performance.
And what if two or more occluding brushes overlap?
You should use the same approach that is used in stencil shadows:
- Clear stencil to 128, clear depth and color
- Render scene to depth buffer (color mask = GL_FALSE)
- Render clipping brushes (all of them) - stencil operation is GL_INCR when z-passes and glCullFace is set to GL_BACK
- Render clipping brushes (all of them) - stencil operation is GL_DECR when z-passes and glCullFace is set to GL_FRONT
- Render scene again with stencil test = 128
Only these pixels that have equal “in” and “out” clipping faces in front of them will be drawn.
Further improvement will include recognizing the case when camera is inside the clipping brush. For every such brush you add 1 to the value you use to clear stencil buffer.
A “perfect” version would require recognizing if any of brush polygons crosses near clip plane inside of view frustum (camera partially inside brush) but this is a rare case, especially with near clip plane very close to 0.
Oh yeah, you could combine steps 3 and 4 by defining separate stencil operations for front and back faces - it can be done on recent GPU’s.
Thanks for your reply…
And what if two or more occluding brushes overlap?
When you say “brushes”, I take it you mean the volumes that clip out parts of some other objects.
I am also unsure what you mean by “overlap”: An overlap of the 2D images on the screen, or intersection of 3D volumes? The latter would be very unlikely in my situation.
If I understand your outline correctly, it certainly seems clever and more general than what I proposed. I hadn’t been thinking in terms of such a global change in the way the whole scene is rendered, but I guess it makes sense. I actually have only certain objects that need to be affected by the clipping volumes, so I think I can avoid rendering the whole scene in your step 2.