Is there a way to determine the ‘facedness’ of a triangle in a vertex shader?
I’m currently doing my geometry in two passes (Cull Front then Cull Back) and setting a shader parameter as -1 or 1. It would be nice to expose the result of the OpenGL pipeline ‘face test’ in the shaders…
You could pass the plane parameters for every polygon and send your position as a shader constant and just do a point-plane check.
[This message has been edited by Humus (edited 02-14-2003).]
It doesn’t make much sense in a vertex shader, does it? A vertex may belong to two different triangles, where one might be front-facing and the other might be back-facing. The best you can do is to test the vertex normal.
For anything more than that, you’d need access to the current triangle to determine facedness. However, triangle setup comes after vertex processing. It’d be a different story if you wanted it in a pixel shader, since that does happen after triangle setup.
Using ARB_vertex_program you can output different colors to be used depending on whether a primitive is front-facing or back-facing. By testing which color makes it to the fragment processing stage, a fragment program could know whether the current primitive is front-facing or back-facing.
– Tom
Originally posted by Tom Nuydens:
[b]It doesn’t make much sense in a vertex shader, does it? A vertex may belong to two different triangles, where one might be front-facing and the other might be back-facing. The best you can do is to test the vertex normal.
For anything more than that, you’d need access to the current triangle to determine facedness. However, triangle setup comes after vertex processing. It’d be a different story if you wanted it in a pixel shader, since that does happen after triangle setup.
Using ARB_vertex_program you can output different colors to be used depending on whether a primitive is front-facing or back-facing. By testing which color makes it to the fragment processing stage, a fragment program could know whether the current primitive is front-facing or back-facing.
– Tom[/b]
I thought that shared vertices would be treated seperately since the actual set-up would be different. Good idea about the fragment colour 
That’s actually what I need - thanks.
For stenciled shadows, the two-sided stencil extensions (e.g., EXT_stencil_two_side) are nice, if available.
There is no current way to get at face information in a fragment program, though Tom’s suggestion is good if you can burn the colors (or at least one component – maybe alpha?). If you need per-fragment alpha, you could try something really sleazy in a vertex program like:
MUL result.color.front.w, alpha, 0.49;
MAD result.color.back.w, alpha, -0.49, 1.0;
Front faces have alpha in the range [0.00,0.49]; back faces are in [0.51, 1.00]. You’d reconstruct the original alpha (with less precision) in the fragment program. You’d have to be careful about the endpoints if you want to preserve alpha values of exactly 0.0 and exactly 1.0.
Note that for vertex programs, you also have to enable VERTEX_PROGRAM_TWO_SIDE to enable front/back-facing color selection – otherwise, you should get only the front color.
Hope this helps,
Pat