Tessellation


OpenGL Rendering Pipeline

Tessellation is the stage in the OpenGL rendering pipeline where patches of vertex data are subdivided into smaller Primitives. This process is governed by two shader stages and a fixedfunction stage.
Contents
Overview
The tessellation process is divided into three stages which form an optional part of the rendering pipeline. Two of the stages are programmable; between them is a fixed function stage. They are described below, in the order they are processed.
Generally, the process of tessellation involves subdividing a patch of some type, then computing new vertex values (position, color, texture coordinates, etc) for each of the vertices generated by this process. Each stage of the tessellation pipeline performs part of this.
The Tessellation Control Shader (TCS) determines how much tessellation to do (it can also adjust the actual patch data, as well as feed additional patch data to later stages). Therefore, the TCS is primarily responsible for ensuring continuity across patches. So if you have two patches that need to have different levels of tessellation, the TCS invocations for the different patches need to use the proper tessellation controls to ensure that the interface between the patches uses the same amount of tessellation. Without this protection, gaps and breaks in what are supposed to be contiguous patches can occur.
The TCS is optional; default tessellation values can be used if no TCS is provided.
The tessellation primitive generator takes the input patch and subdivides it based on values computed by the TCS or provided as defaults.
The Tessellation Evaluation Shader (TES) takes the tessellated patch and computes the vertex values for each generated vertex.
Patches
Tessellation stages operate on patches, a primitive type denoted by the constant GL_PATCHES. A patch primitive is a generalpurpose primitive, where every n vertices is a new patch primitive. The number of vertices per patch can be defined on the applicationlevel using:
void glPatchParameteri(GLenum pname, GLint value);
with GL_PATCH_VERTICES as target and a value which has is on the halfopen range [1, GL_MAX_PATCH_VERTICES]. The maximum number of patch vertices is implementationdependent, but will never be less than 3.
Tessellation control shader
The first step of tessellation is the optional invocation of a tessellation control shader (TCS). The TCS has two jobs:
 Determine the amount of tessellation that a primitive should have.
 Perform any special transformations on the input patch data.
The TCS can change the size of a patch, adding more vertices perpatch or providing fewer. However, a TCS cannot discard a patch (directly; it can do so indirectly), nor can it write multiple patches. Therefore, for each patch provided by the application, one patch will be provided to the next tessellation stage.
The TCS is optional. If no TCS is active in the current program or program pipeline, then the patch data is passed directly from the Vertex Shader invocations to the tessellation primitive generation step. The amount of tessellation done in this case is taken from default values set into the context. These are defined by the following function:
void glPatchParameterfv(GLenum pname, const GLfloat *values);
When pname is GL_PATCH_DEFAULT_OUTER_LEVEL, values is a 4element array of floats defining the four outer tessellation levels. When pname is GL_PATCH_DEFAULT_INNER_LEVEL, values is a 2element array of floats defining the two inner tessellation levels.
These default values correspond to the TCS perpatch output variables gl_TessLevelOuter[4] and gl_TessLevelInner[2].
Tessellation primitive generation
Primitive generation is a fixedfunction stage responsible for creating a set of new primitives from the input patch. This stage is only executed if a tessellation evaluation shader (TES) is active in the current program or program pipeline. Primitive generation is affected by the following factors:
 The tessellation levels, provided either by the TCS or the default values, as stated above.
 The spacing of the tessellated vertices, as defined by the subsequent TES stage. It may be equal_spacing, fractional_even_spacing, or fractional_odd_spacing.
 The input primitive type defined by the subsequent TES which may be one of triangles, quads or isolines. The TES can also force the generation of the tessellation as a series of points rather than triangles or lines by providing the point_mode primitive.
 The primitive generation order defined by the subsequent TES, which may be cw or ccw.
Abstract patch
Notice that the primitive generation is not affected by the outputs of the TCS, the TCS's output patch size, any perpatch TCS outputs (besides the tessellation levels), and so forth. The primitive generation part of the tessellation stage is completely blind to the actual vertex coordinates and other patch data.
The purpose of the primitive generation system is to determine how many vertices to generate, in which order to generate them, and what kind of primitives to build out of them. The actual pervertex data for these vertices, such as position, color, etc, is to be generated by the TES, based on information provided by the primitive generator.
Because of this dichotomy, the primitive generator operates on what could be considered an "abstract patch". It doesn't look at the patch output from the TCS; it thinks only in terms of tessellating an abstract quad, triangle, or "isoline" block.
Depending on the abstract patch type, the primitive generator evaluates a different number of tessellation levels and applies different tessellation algorithms. Each generated vertex has a normalized position (i.e. the coordinates are on the range [0, 1]) within the abstract patch. This position has two or three components, depending on the type of the patch. The coordinates are provided to the TES via the builtin in vec3 gl_TessCoord input.
Tessellation levels
The amount of tessellation that is done over the abstract patch type is defined by inner and outer tessellation levels. These, as previously stated, are provided either by the TCS or by context parameters specified via glPatchParameter. They are a 4vector of floats defining the "outer tessellation levels" and a 2vector of floats defining the "inner tessellation levels."
The specific interpretation depends on the abstract patch type being used, but the general idea is this. In most cases, each tessellation level defines how many segments an edge is tessellated into; so a tessellation level of 4 means that an edge will become 4 edges (2 vertices become 5). The "outer" tessellation levels define the tessellation for the outer edges of the primitive. This makes it possible for two or more patches to properly connect, while still having different tessellation levels within the patch. The inner tessellation levels are for the number of tessellations within the abstract patch.
Not all abstract patches use the same number of values in the outer/inner tessellation levels data. For example, triangles only uses one inner level and 3 outer levels. The rest are ignored.
The tessellation levels specified in this way are not directly used. They go through a clamping process to generate the effective tessellation levels that are used to tessellate the primitive. This process depends on the TES's spacing parameter.
In the below discussion, max is the maximum allowed tessellation level, as defined by the GL_MAX_TESS_GEN_LEVEL. It must be at least 64, so you have some room to play with.
The spacing affects the effective tessellation level as follows:
 equal_spacing
 Each tessellation level is individually clamped to the closed range [1, max]. Then it is rounded up to the nearest integer to give the effective tessellation level.
 fractional_even_spacing
 Each tessellation level is individually clamped to the closed range [2, max]. Then it is rounded up to the nearest even integer to give the effective tessellation level.
 fractional_odd_spacing
 Each tessellation level is individually clamped to the closed range [1, max  1]. Then it is rounded up to the nearest odd integer to give the effective tessellation level.
The patch can be discarded if any outer tessellation level that is used by the abstract patch type is 0 or less. It can also be discarded if it is a floatingpoint NaN. A patch that is discarded does not get tessellated, and no TES is invoked for it. It is simply swallowed by the system as though it never were.
This allows a TCS to effectively cull patches by passing 0 for a relevant outer tessellation level.
Edge tessellation spacing
At various points in the forthcoming discussion about tessellating the abstract patch, there will be statements that say to tessellate an edge of some primitive. This means to subdivide it into a series of segments according to some tessellation level. The total length of these segments is the length of the original segment. However, the length of individual segments depends on the spacing parameter specified in the TES.
Given an effective tessellation level, denoted by n, which applies to that edge, the vertices for an edge tessellated by n is defined as:
 equal_spacing
 The edge is divided into n segments. All segments will have equal length.
 fractional_even_spacing, fractional_odd_spacing
 If n is 1, then no subdivision occurs. Otherwise, the edge will be divided into n  2 segments of equal length. There will also be 2 segments that have length equal to each other, but not necessarily to the first group. The length of these 2 segments, relative to the others, will be n  f, where f is the effective tessellation level value after clamping but before being rounded up.
 When n == f, the length of the 2 segments will be equal to the length of the other segments. As n  f approaches 2.0, the relative length of the 2 segments approaches 0.0.
 The exact location of the 2 shorter segments is not defined, but they should be placed symmetrically, on opposite sides of the subdivided edge. Also, the location must be invariant with the same f value (thus allowing tessellated edges to work together).
The purpose of the fractional spacing modes is to have smoother, more stable interpolation as tessellation levels change. This is best used if tessellation levels are based on the distance to the camera or something.
Tessellating primitives
This section is missing information. Further details can be found on the talk page. 
Triangles
The abstract patch of the triangle tessellation is a triangle, naturally. Only the first three outer tessellation levels are used, and only the first inner tessellation level is used.
Each vertex generated and sent to the TES will be provided Barycentric coordinates as the gl_TessCoord input. This defines where this vertex is located within the abstract triangle. With this coordinate, it is possible to multiply any vertex attribute from 3 vertices to compute the appropriate value from the tessellation unit.
Quads
The abstract patch of a quad is a rectangle, naturally. All 4 outer and 2 inner tessellation levels are used.
Each vertex generated and sent to the TES will be provided a normalized 2D coordinate as the gl_TessCoord input, representing the location of that vertex within the abstract patch.
Isolines
The abstract patch of isolines is a rectangle, oddly enough. Only the first two outer tessellation levels are used; none of the inner tessellation levels are used.
The rectangular abstract isolines patch represents a series of horizontal lines. The first outer tessellation level defines how many segments the lines are divided into, and the second outer level defines how many lines are in the patch. So if you want to tessellate a single line, you should pass 1 for gl_TessLevelOuter[1]. However, if you need more line tessellation than can be done in a single outer level value, the TES is perfectly capable of stitching multiple separate lines together. Remember: the abstract patch is abstract; the TES can position the vertices wherever it wants.
Each vertex generated and sent to the TES will be provided a normalized 2D coordinate as the gl_TessCoord input. The y component specifies which line (normalized to the halfopen range [0, 1) ) is being generated. The x component defines how far along that line this vertex should be generated for.
Tessellation evaluation shader
The Tessellation Evaluation Shader (TES) is responsible for taking the abstract coordinates generated by the primitive generator, along with the outputs from the TCS (or vertex shader, if no TCS is used), and using them to compute the actual values for the vertices. This is where you code the algorithm that you actually use to compute the new positions/normal/texcoords/etc. The TES is a mandatory part of tessellation; if one is not present, then tessellation doesn't happen.
The TES is rather like a vertex shader, in that each invocation operates on a distinct vertex within the tessellated patch (though, as with a vertex shader, the system may call the TES more than once for the same vertex, so it should be deterministic). Also, the TES cannot cull vertices.