glStencilOp's GL_INCR & GL_DECR behaviour when masked?

Hi everybody!
What happens to a stencil index value in the stencil buffer when I specify GL_INCR or
GL_DECR as the stencil operations to do with glStencilOp() for any of the stencil tests,
if I call glStencilMask() with a mask value different than all 1’s? Does the stencil
index gets just incremented/decremented? Or the value is gathered, incremented/decremented,
and then masked to be written again to the stencil buffer? The OpenGL specification is
not very clear (at least for me) on this, so I hope anyone of you can answer me that.

The OpenGL specifications is very clear about it.
All stencil operations behave in the same way: thay work with the stencil values, not some masked/gathered values.

The glStencilMask function controls the writing of individual bits in the stencil planes.

The mask in glStencilFunc( function, reference_value, mask ) - for stencil comparison only, not for stencil operations.

Stencil test

Thanks Serge for your quick response. Sorry if I answered until now, but it was until
now when I came to work again.

Serge, I think you misunderstood me. I know that the “mask” parameter in
glStencilFunc(function, reference_value, mask) is used for stencil comparisons.
In fact, I wasn’t talking about glStencilFunc() at all; it’s about
glStencilOp ( enum sfail, enum dpfail, enum dppass ), where the stencil buffer operations
are actually specified for each separate stencil/depth test. Specifically, what I
wanted to know is what happens to the stencil values when GL_INCR and/or GL_DECR
are specified AND a value distinct from “111111111111…1” is specified with
glStencilMask().

An example:
A fragment at (50,50) is about to be renderized at the framebuffer. The 16-bit
stencil buffer value at (50,50) is 0x003E.
The GL is told to GL_INCR the stencil values
when the depth test fails. Additionally, glStencilMask(0xFFEA) was called. The depth
test fails for the fragment at (50,50). What will be the new value of the stencil
buffer at (50,50) (assuming there are no more operations applied to the stencil
buffer)?

The glStencilMask function controls the writing of individual bits in the stencil planes.

StencilBits = (StencilBits & ~StencilMask) | (StencilOp(…) & StencilMask);

I just recently bumped into this problem so I’d like to add my two cents in case anyone else is interested…

AFAIK, opengl stencil masks are not very friendly with GL_INCR, GL_DECR (and GL_INCR_WRAP, GL_DECR_WRAP).

OGL stencils have two related masks:

  • glStencilMask() which is applied before writing to stencil
  • glStencilFunc() mask which is only used for the stencil test

As previously stated, glStencilOp() function acts on the original stencil value without any masks.

Therefore it is impossible to increment only a submasked value of the stencil with GL_INCR or GL_DECR (unless it includes the lowest bits, as explained below). An arbitrary submask would require extra shifts before/after incrementing/decrementing the stencil value.

As an example, I do not think there is any way to use only the upper bits (e.g. glStencilMask(0xf0)) of the stencil for stencil shadows (which use GL_INCR and GL_DECR).

However, it is possible to use the lower bits (e.g. glStencilMask(0x0f)) with GL_INCR_WRAP/GL_DECR_WRAP because no shifts are necessary before/after incrementing/decrementing the stencil value. By trying examples on paper, you can see that overflow/underflow wraps correctly.

Hope this helps.