Stencil Test

OpenGL Rendering Pipeline

Rendering Pipeline Overview

---

• Vertex Specification
  • Vertex Rendering
  • Primitive
• Vertex Processing
  • Vertex Shader
  • Tessellation
  • Geometry Shader
• Vertex Post-Processing
  • Transform Feedback
• Primitive Assembly
  • Face Culling
• Rasterization
• Fragment Shader
• Per-Sample Processing
  • Scissor Test
  • Stencil Test
  • Depth Test
  • Blending
  • Logical Operation
  • Write Mask

The Stencil Test is a per-sample operation performed after the Fragment Shader. The fragment's stencil value is tested against the value in the current stencil buffer; if the test fails, the fragment is culled.

Stencil buffer

In order to use the stencil test, the current Framebuffer must have a stencil buffer. The stencil buffer is an image that uses a stencil image format. The Default Framebuffer may have a stencil buffer, and user-defined framebuffers can attach stencil formatted images (either depth/stencil or stencil-only) to the GL_STENCIL_ATTACHMENT attachment point.

If the current framebuffer has no stencil buffer, then the stencil test will always behave as if it is disabled.

If there is a stencil buffer, that buffer has a certain bitdepth. This specifies the number of stencil bits available.

Fragment stencil value

Each Fragment has a stencil value, which is an unsigned integer. The stencil test operation will test this stencil value against the value from the current framebuffer at the fragment's position.

This value is usually defined by the same function that sets the stencil test, below. This means that every fragment from every primitive for every object in a draw call gets the same fragment stencil value (with the front/backface point noted below).

Stencil test

To enable stencil testing, call glEnable with GL_STENCIL_TEST. When the stencil test is disabled, no testing happens and the stencil value in the framebuffer is not modified.

When rendering to a framebuffer that has no a stencil buffer, stenciling will always behave as if it is disabled.

Stenciling operations take into account the fact that triangles have two sides. Therefore, all stencil tests and operation functions have two sets of data: one for the front side of triangles and one for the back. Which side is used depends on whether the fragments generated from the primitive came from the front or back face.

These functions all take a face​ parameter, which specifies which facing state is set by that function. The face​ can be GL_FRONT or GL_BACK, but it can also set both sides at once with GL_FRONT_AND_BACK.

For Primitives that have no facing, the front side stencil state are always used.

The stencil test itself is set by this function:

void glStencilFuncSeparate(GLenum face​, GLenum func​, GLint ref​, GLuint mask​);

The ref​ defines the fragment's stencil value for all fragments generated for the given facing. The fragment stencil value will be clamped to the range defined by the stencil buffer's bitdepth.

The first step of the stencil test is to get the destination stencil value from the stencil buffer (called Ds). This is an unsigned integer value. The fragment's stencil value will be called Fs.

The next step is to perform a bitwise AND with both Fs and Ds, against the mask​ parameter for both. This allows the user to mask off certain stencil bits, reserving them for different conditional tests. This results in two masked unsigned integers, Fm and Dm.

Then the stencil test itself is performed between Fm and Dm, based on the func​ parameter. The test is of the form (Fm FUNC Dm); the masked fragment value is on the left-hand side. The available functions are:

Enum	Test	Enum	Test
GL_NEVER	Always fails.	GL_ALWAYS	Always passes
GL_LESS	<	GL_LEQUAL	≤
GL_GREATER	>	GL_GEQUAL	≥
GL_EQUAL	=	GL_NOTEQUAL	≠

glStencilFunc can be used to set both face operations in one call, if you don't like to use GL_FRONT_AND_BACK.

Stencil operations

The stencil operation uses the result of the stencil test to decide how to modify the stencil value in the framebuffer. If stencil testing is disabled, then this step is skipped and framebuffer's stencil value is unmodified.

If the stencil test fails, the fragment is discarded. Normally, discarding a fragment means that it has no visible effects. However, the stencil buffer can be updated even from discarded fragments, if the depth or stencil tests discard them.

Note: Recall that the stencil and depth tests are normally the last per-sample processing steps that can discard fragments. However, if fragment tests happen early, even if the fragment shader discards the fragment, the stencil buffer can be modified.

There are three possible cases involved here:

• The stencil test fails. The fragment will be discarded. Remember that the stencil test happens before the depth test, so the result of that is never computed if the stencil test fails.
• The stencil test passes, but the Depth Test fails. The fragment will be discarded (due to the depth failure). This case only applies if the depth test is enabled.
• The stencil test passes, and the Depth Test passes. If the depth test is disabled, then it is always assumed to pass.

To define how the stencil value in the framebuffer is modified in each case, use the following function:

void glStencilOpSeparate(GLenum face​, GLenum sfail​, GLenum dpfail​, GLenum dppass​);

The parameters sfail​, dpfail​, and dppass​ define the stencil update operations to perform in the three cases above, respectively. Each case can use any of the following operations ("current value" here means the value already in the stencil buffer):

Enum	Operation	Enum	Operation
GL_KEEP	Don't modify the current value (default)	GL_INVERT	Invert the current value
GL_ZERO	Set it to zero	GL_REPLACE	Replace with the masked fragment value
GL_INCR	Increment the current value, saturating1 if it would overflow	GL_INCR_WRAP	Increment the current value, wrapping if it would overflow
GL_DECR	Decrement the current value, setting to zero if it would underflow	GL_DECR_WRAP	Decrement the current value, wrapping if it would underflow

¹: Meaning that it stops at the maximum representable integer at the stencil buffer's bitdepth. For an 8-bit stencil buffer, that would be 255.

glStencilOp can be used to set both face operations in one call, if you don't like to use GL_FRONT_AND_BACK.

See also

• Depth Test
• Fragment Shader