FAQ/Stencil Mask

How do I render to a nonrectangular viewport?

OpenGL's stencil buffer can be used to mask the area outside of a non-rectangular viewport. With stencil enabled and stencil test appropriately set, rendering can then occur in the unmasked area. Typically an application will write the stencil mask once, and then render repeated frames into the unmasked area.

As with the depth buffer, an application must ask for a stencil buffer when the window and context are created.

An application will perform such a rendering as follows:

  /* Enable stencil test and leave it enabled throughout */
  glEnable(GL_STENCIL_TEST);

  /* Prepare to write a single bit into the stencil buffer in the area outside the viewport */
  glStencilFunc(GL_ALWAYS, 0x1, 0x1);

  /* Render a set of geometry corresponding to the area outside the viewport here */

  /* The stencil buffer now has a single bit painted in the area outside the viewport */

  /* Prepare to render the scene in the viewport */
  glStencilFunc(GL_EQUAL, 0x0, 0x1);

  /* Render the scene inside the viewport here */

  /* ...render the scene again as needed for animation purposes */

After a single bit is painted in the area outside the viewport, an application may render geometry to either the area inside or outside the viewport. To render to the inside area, use glStencilFunc(GL_EQUAL,0x0,0x1), as the code above shows. To render to the area outside the viewport, use glStencilFunc(GL_EQUAL,0x1,0x1).

You can obtain similar results using only the depth test. After rendering a 3D scene to a rectangular viewport, an app can clear the depth buffer and render the nonrectangular frame.