PDA

View Full Version : Understanding stencil buffer and masking



DragonautX
04-12-2017, 11:33 PM
In the Stencil Testing (https://learnopengl.com/#!Advanced-OpenGL/Stencil-testing) tutorial on Learn OpenGL, I'm having trouble understanding how the use of glStencilFunc() and glStencilMask() can be used draw only around the borders of an object (like a highlighting effect).

From what I know about stencil buffers so far, a stencil buffer with only 0s could look like this on a 4x4 pixel screen, with the given drawing code:



glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilMask(0xFF); //0xFF would be new stencil value on stencil test success

glDisable(GL_DEPTH_TEST); //would also mean depth test always passes
glDrawArrays(<center 4 pixels>);


0000 0000 0000 0000
0000 0xFF 0xFF 0000
0000 0xFF 0xFF 0000
0000 0000 0000 0000

I'd be interested in drawing a different color object on the pixels with only a "0000" value in the stencil buffer. Based on the tutorial, I could just call glDrawArrays() to draw across all the pixels, and the stencil buffer code below should discard the center 4 pixel fragments.



glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
glStencilMask(0x00); //on stencil success or fail, no stencil values will be changed

glDisable(GL_DEPTH_TEST);
glDrawArrays(<draw to the entire 4x4 pixel screen>);


Doing glStencilFunc for a stencil value of 0xFF in the stencil buffer doesn't seem to discard the fragment though when I think of the math behind it (though the code output in the tutorial shows discarding). From the docs to glStencilFunc, the stencil test would be like this with the arguments above:

func=GL_NOTEQUAL, ref=1, mask=0xFF, stencil=0xFF (since the stencil value is 0xFF)
(ref & mask) != (stencil & mask) ---> (1 & 0xFF) != (0xFF & 0xFF) ---> 1 != 0xFF ---> stencil test passes

I feel like this would make sense and the fragments would be drawn where the stencil value is 0xFF, but the tutorial output shows pixels being drawn like only where the stencil buffer values are 0 I think (hence the highlighting effect). Is there something wrong with my logic with the stencil buffer, maybe in the first or second drawing code? I feel like I summarized the code right from the tutorial.

muthuveera
04-13-2017, 04:33 AM
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

In this, GL_REPLACE(Stencil pass & Depth pass) will replace old stencil value to Reference value.

glStencilFunc(GL_ALWAYS, 1, 0xFF);
Ref Value: 1
Mask: 0xFF

so after rendering your first fragment your stencil plane will become:

0x00 0x00 0x00 0x00
0x00 0x01 0x01 0x00
0x00 0x01 0x01 0x00
0x00 0x00 0x00 0x00

Then Second fragment calculation:
(ref & mask) != (stencil & mask) ---> (1 & 0xFF) != (1 & 0xFF) ---> 1 != 1 ---> stencil test fails.

DragonautX
04-13-2017, 09:20 AM
That makes sense. Looks like I got the first drawing code wrong. The docs in glStencilOp say GL_REPLACE means replace the stencil value with ref, like you said. Thanks!