Stencil Buffer Shenanigans

I’m working on a stencilling system for my latest project that works like this:

A function call, BeginStencilDraw() will set all color masks to false and the stencil mask to true. As a result, all subsequent drawing commands will create stencilling areas. EndStencilDraw() is the inverse function that will return the color mask and disable the stencil mask.

There is also a counter variable that is initially zero, that corresponds to the ref value for the stencil test. This is the clear value for the stencil buffer, so if the counter is at zero then the stencil test is disabled for performance. BeginStencilDraw() will enable the stencil test if the counter is at zero, because it also uses glStencilOp(GL_KEEP, GL_KEEP, GL_INCR) so that the pixels that should become part of the stencil area are appropriately promoted to the next stencil value. EndStencilDraw() sets glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP) and disables the test if the counter is at zero.

The other functions are EnterStencil() which increments the counter, calls glStencilFunc(GL_EQUAL, ++Counter, 0xff) so that the stencil areas previously drawn are used. If the counter is zero on the way into the function, the stencil test is enabled first. Its companion function ExitStencil() willc all glStencilFunc(GL_EQUAL, --Counter, 0xff) and disable the stencil test if counter is zero. There is also rollover protection in that EnterStencil() will abort if the stencil counter is 255 (8-bit stencil buffer, more on this later) and ExitStencil() will abort if the stencil value is already 0.

The idea is that this will help me nest stencil areas easily, as such:

if(Transition)
{
BeginStencilDraw();
DrawAnAreaOnScreenForAWipeEffect();
EndStencilDraw();
EnterStencil();
}
BeginStencilDraw();
DrawAnAreaThatWillBeStencilledInTheScene();
EndStencilDraw();

DrawScene();

EnterStencil();
DrawTheMirrorImageOnTheWall();
ExitStencil();

if(Transition)
ExitStencil();

The problem, however, is that the StencilOp is not being performed properly. If I attempt to draw another stencil area inside an area of stencil value one, instead of incrementing the stencil value will get zeroed. I have checked this by dumping images of the stencil buffer and examining them. It’s not precisely zeroed as it is decremented. If I use a default nonzero stencil value and clear value, such as 4, the first area is correctly drawn with value 5, but when I EnterStencil and draw the next stencil area the second area gets a value of 4. I can provide more code upon request, but this problem has me frustrated.

The platform is Win32 on an nVidia Geforce 3, but I have duplicated the problem on a friend’s Geforce 2 GTS and another’s ATI Radeon. My pixel format calls for:

ColorBits 32
DepthBits 16
StencilBits 8

At first I thought that since the stencil values went 0, 1, then back to 0 that I was only getting a 1 bit stencil buffer, but I examined my pixel format with DescribePixelFormat() and it surely says 8, and the test I did with initial values >1 dismissed that theory.

Also, if I leave the stencil test on the whole time, the stencil buffer gets completely messed up; it fills up with tons of artifacts and junk data. I can’t figure this one out.