In the Stencil Testing tutorial on Learn OpenGL, I’m having trouble understanding how the use of glStencilFunc() and glStencilMask() can be used draw only around the borders of an object (like a highlighting effect).
From what I know about stencil buffers so far, a stencil buffer with only 0s could look like this on a 4x4 pixel screen, with the given drawing code:
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilMask(0xFF); //0xFF would be new stencil value on stencil test success
glDisable(GL_DEPTH_TEST); //would also mean depth test always passes
glDrawArrays(<center 4 pixels>);
0000 0000 0000 0000
0000 0xFF 0xFF 0000
0000 0xFF 0xFF 0000
0000 0000 0000 0000
I’d be interested in drawing a different color object on the pixels with only a “0000” value in the stencil buffer. Based on the tutorial, I could just call glDrawArrays() to draw across all the pixels, and the stencil buffer code below should discard the center 4 pixel fragments.
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
glStencilMask(0x00); //on stencil success or fail, no stencil values will be changed
glDisable(GL_DEPTH_TEST);
glDrawArrays(<draw to the entire 4x4 pixel screen>);
Doing glStencilFunc for a stencil value of 0xFF in the stencil buffer doesn’t seem to discard the fragment though when I think of the math behind it (though the code output in the tutorial shows discarding). From the docs to glStencilFunc, the stencil test would be like this with the arguments above:
func=GL_NOTEQUAL, ref=1, mask=0xFF, stencil=0xFF (since the stencil value is 0xFF)
(ref & mask) != (stencil & mask) —> (1 & 0xFF) != (0xFF & 0xFF) —> 1 != 0xFF —> stencil test passes
I feel like this would make sense and the fragments would be drawn where the stencil value is 0xFF, but the tutorial output shows pixels being drawn like only where the stencil buffer values are 0 I think (hence the highlighting effect). Is there something wrong with my logic with the stencil buffer, maybe in the first or second drawing code? I feel like I summarized the code right from the tutorial.