I haven’t gotten into tessellation yet, but have some questions. How does the algorithm handle edges? If I have a 6 sided box with hard edges, won’t that cause gaps to appear along the edges, if the surface is just projected straight out of the normal?
How does the algorithm handle edges?
How does what algorithm handle edges? There are many, many tessellation algorithms. Some have special cases for edges, and some do not.
Also, which from of tessellation are you referring to? Are you talking about the old GLU tessellators, geometry shader tessellation, or OpenGL 4.x-style tessellation?
I am talking about OpenGL 4 tessellation, the only useful one.
OpenGL 4.x tessellation is controlled by you, the programmer. There are three OpenGL 4.x pipeline stages that implement tessellation: first is the tessellation control shader (if present, executes immediately after the vertex shader), next in the GL pipeline is the tessellation primitive generator (executes only if there is a tessellation evaluation shader), followed in the GL pipeline by the tessellation evaluation shader. (The geometry shader, if present, is next in the GL pipeline.)
If there is a tessellation control shader, its principle intended function is to dynamically control the number of tessellation divisions, but not how those divisions are actually performed. Your tessellation control shader program does this by defining the elements of the gl_TessLevelOuter[] and gl_TessLevelInner[] arrays. If there is no tessellation control shader, then you control the number of tessellation divisions with your application code via the glPatchParameterfv() function. In either case, you must define as many elements in gl_TessLevelOuter[] as there are edges in the primitive being drawn, that is there’s one element per edge.
The tessellation evaluation shader controls how the tessellation primitive generator performs the divisions that the tessellation control shader (or glPatchParameterfv()) specified. There are three options, and they are specified with the tessellation evaluation shader layout qualifier: equal_spacing, fractional_even_spacing, and fractional_odd_spacing.
That’s the background information. There’s one more piece to the puzzle: OpenGL 4.0 (at least) has eight rules of tessellation invariance. (See section A.4 in the OpenGL 4.0 core specification.) These define how tessellation must be performed by the GL implementation.
Getting back to your question directly, to prevent gaps from occurring between edges of adjacent tessellated primitives, you need to understand the rules of tessellation invariance and write your code appropriately. But in a nutshell, there’s a really easy take-away from all this: as long as adjacent primitives share identical edge definitions for their common edges (i.e., share the same vertices), and are tessellated with an identical number of divisions for each primitives’ shared edges, and with identical layouts, then they will be tessellated identically and there will be no gaps between them. The OpenGL 4.0 rules of tessellation invariance guarantee it.
They may be tessellated the same, but unless they have the same texture coordinates, they will not be displaced the same. In my example of a 6-sided box with hard edges, is there a way to write the displacement shader so that vertices along the edges have 0 displacement depth, and vertices inside a surface have >0 displacement depth? What if the box is made up of triangles? How can you determine that each square face’s outer edges should have 0 displacement, and the inner edge between the two triangles making up the face should have >0 displacement?
They may be tessellated the same, but unless they have the same texture coordinates, they will not be displaced the same.
The part you don’t understand is that you (specifically, your tessellation evaluation shader) explicitly controls the position of the tessellated vertices. The algorithm for “displacing” a vertex is something you write.
You can accomplish what you want without any difficulty.
First, some more background information:
The tessellation shaders work on a new primitive type, GL_PATCHES, that is exclusively for tessellation, and the tessellators only work with GL_PATCHES primitives. A GL_PATCHES primitive consists of anywhere from one to 32 vertices (all the patches need the same number of vertices, you decide how many). The vertices per patch have no inherent structure or meaning to OpenGL; their use and purpose is determined by whatever convention you ascribe to them, so they are really points rather than vertices, but they are each transformed by the vertex shader before the tessellator stage, and each of them has its own vertex attributes (such as texture coordinates, for example). From what you’re describing, you’d use one vertex each to define the four corners of each of your box’s six sides. In other words, four vertices per patch.
The layout qualifier of the tessellation evaluation shader specifies how the tessellation primitive generator is configured. There are three possible configurations: triangles, quads, and isolines. What this refers to is what and how the tessellation primitive generator performs tessellation. The tessellation primitive generator outputs normalized parametric coordinates. It stores them in the built-in vector variable gl_TessCoord: (u, v) coordinates for quads, (u, v, w) barycentric coordinates for triangles, and (u, v) coordinates for isolines (isolines are a variation on quads). In all cases, 0.0 <= u, v, w <= 1.0, and in the case of triangles only, u + v + w = 1.0 (i.e., they’re barycentric).
All of this stuff comes together in and is interpreted by your tessellation evaluation shader. The tessellation evaluation shader is executed once for each (u, v) (quad or isoline layout) or (u, v, w) (triangle layout) point generated by the tessellation primitive generator and stored in gl_TessCoord.
That’s the generalized background info; I made the description more complicated by describing all the layout options (quads, triangles, and isolines) rather than just describing the quad layout.
Getting back to your questions: You asked how can you determine that each square face’s outer edges should have zero displacement, and the inner edge between the two triangles making up the face should have >0 displacement? There are two ways I know of.
One way uses quads for your patch layout. If so, your tessellation evaluation shader will know when it is at the edge of the side of the box whenever u or v equals zero or one. Your tessellation evaluation shader doesn’t even have to make any special consideration regarding the inner edge between the two triangles making up the face; in general, there won’t be two triangles making up the face (that depends on the values that you stored in gl_TessLevelOuter[] and gl_TessLevelInner[] either in the tessellation control shader or with the glPatchParameterfv() function).
The other way depends on your displacement texture map to know where the “hard” edges are, and to specify zero displacement along them. This method doesn’t care whether your layout is quads or triangles.
You cannot use triangles for your patches and layout and “dumb” (i.e., not knowing where the hard edges are) displacement texture maps. Though your tessellation evaluation shader will know when it is at the edge of any of your triangle patches whenever u, v, or w is zero (that’s how barycentric coordinates work with triangles), there is no way to determine whether you are on an edge of your box face, or on the inner edge of one of your pair of triangle patches per face.
This might sound really complicated, but it’s not bad. The key thing is knowing what each of the three tessellation stages in the GL pipeline does and what inputs they have and what they output.
One of the things that might make this seem more confusing is the connection between the patch vertices and quad, triangle, and isoline layouts. Why can there be up to 32 vertices per patch if you can’t have a layout with more than four sides? Because you can do anything you want with the vertices; they’re really points and not necessarily vertices. The layout refers to parametric space, not world space. The vertices/points live in some world space. If you are going to use only a displacement texture map for controlling the shape of your geometry, then you will always use three vertices per triangle or four vertices per quad or two or four vertices per isoline patch, and a matching layout. But if you are using (or also using) some type of spline to control the shape of your geometry, then you may need some supplemental points. For example, a cubic Bezier curve requires four points per patch to define an isoline layout, and a bicubic Bezier surface patch requires 16 points to define a quad (or isoline) layout.
In any case your tessellation evaluation shader takes all a patch’s vertices along with their attributes and the gl_TessCord normalized parametric coordinates in and evaluates them however you have programmed it to to define one output point and its attributes (if any). Typically, you would use the parametric coordinates to interpolate between the patch vertices (and attributes, if appropriate, such as with texture coordinates). Following the tessellation evaluation stage, its outputs are assembled into triangles (for quad or triangle layouts) or into lines (for isoline layouts) and sent to the geometry shader (if present) or rasterized if there is no geometry shader.
Thanks for the explanation. I just wanted to gather some information before I actually dive into it. That helps a lot.
I think I have a way to make this work for CSG-style geometry, with arbitrary texcoords and displacement maps, and faces with different numbers of sides.