The stencil test conditionally discards a fragment based on the outcome of a comparison between the value in the stencil buffer at location and a reference value. The test is controlled with
void StencilFunc ( enum func, int ref, uint mask ) ;
void StencilOp ( enum sfail, enum dpfail, enum dppass ) ;
The test is enabled or disabled with the Enable and Disable commands, using the symbolic constant STENCIL_TEST. When disabled, the stencil test and associated modifications are not made, and the fragment is always passed.
ref is an integer reference value that is used in the unsigned stencil comparison. It is clamped to the range , where s is the number of bits in the stencil buffer. func is a symbolic constant that determines the stencil comparison function; the eight symbolic constants are NEVER, ALWAYS, LESS, LEQUAL, EQUAL, GEQUAL, GREATER, or NOTEQUAL. Accordingly, the stencil test passes never, always, if the reference value is less than, less than or equal to, equal to, greater than or equal to, greater than, or not equal to the masked stored value in the stencil buffer. The s least significant bits of mask are bitwise ANDed with both the reference and the stored stencil value. The ANDed values are those that participate in the comparison.
StencilOp takes three arguments that indicate what happens to the stored stencil value if this or certain subsequent tests fail or pass. sfail indicates what action is taken if the stencil test fails. The symbolic constants are KEEP, ZERO, REPLACE, INCR, DECR, and INVERT. These correspond to keeping the current value, setting it to zero, replacing it with the reference value, incrementing it, decrementing it, or bitwise inverting it. For purposes of increment and decrement, the stencil bits are considered as an unsigned integer; values clamp at 0 and the maximum representable value. The same symbolic values are given to indicate the stencil action if the depth buffer test (below) fails ( dpfail), or if it passes ( dppass).
If the stencil test fails, the incoming fragment is discarded. The state required consists of the most recent values passed to StencilFunc and StencilOp , and a bit indicating whether stencil testing is enabled or disabled. In the initial state, stenciling is disabled, the stencil reference value is zero, the stencil comparison function is ALWAYS, and the stencil mask is all ones. Initially, all three stencil operations are KEEP. If there is no stencil buffer, no stencil modification can occur, and it is as if the stencil tests always pass, regardless of any calls to StencilOp .