“Draw shape where stencil buffer==x, if passes, replace buffer value with y”
No, you can’t do that. Not with stencil testing. The stencil operation that writes the stencil value only deals with two values: the current value from the stencil buffer, or the reference provided via glStencilFunc. You can’t provide a third value to this process (unless y is always zero).
That being said, the overall goal is possible with the stencil test. It just requires a bit of work.
First, clear the depth buffer to 0. Also, initialize an integer to 0. Let’s refer to this integer as ref. Also, enable the stencil test.
When you go to render any root object, do the following (passing all root objects a ref value of 0):
- Set up your stencil test so that the stencil test passes if the stencil value is exactly <= to
ref. And if it does pass, you want to increment the stencil value. Use this code:
glStencilFunc(GL_LEQUAL, ref, 0xFFFFFFFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
-
Render the object.
-
Call every child object’s rendering function, but the ref value you pass to them should be ref + 1. Each object at the same level of the hierarchy should get the same ref value (I know what you’re thinking, and it’s covered). This will obviously be recursive.
-
Here’s the part that makes everything work. You now need to revert your stencil buffer back to where it was before this function was call. To do that, you want to re-render this object, without writing any colors (or depth values). But you do want to update the stencil value, changing it back to this function’s version of ref.
However, you don’t want the stencil test to always pass; you only want the stencil to be reset where the stencil value was changed by your object’s rendering or any child objects’ rendering. So you do this:
glWriteMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); //Turn off color writing.
glDepthMask(GL_FALSE); //Turn off depth writing.
glStencilFunc(GL_GEQUAL, ref, 0xFFFFFFFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
-
Render this object again.
-
Turn color/depth masking back off:
glWriteMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); //Turn on color writing.
glDepthMask(GL_TRUE); //Turn on depth writing.
This method will work so long as the depth of your hierarchy of objects does not exceed the maximum value in your depth buffer (for 8 bits, 255).
Then again, you could do something similar with the depth buffer, so long as you can control the depth at which each shape gets rendered without modifying its mesh data.
