The two pass solution is what I do now, but if the fragment shader is heavy, it is not a happy place.
Roughly speaking the idea is a follows: when one runs the fragment shader, one can compute a coverage value. Now, if that coverage is not “enough”, then one would like to draw that fragment blended with the background [I am not talking about transparent stuff at all]. That bites as then that requires rendering back to front… so, break it into two separate buffers:
[ol]
[li]initialization: setup 2 target FBO with RGBA(opaque color), RGBA(transparent)[/li][li]set opaque target to blending add with operation being GL_SRC_ALPHA and GL_ONE_MINUS_SRC_ALPHA[/li][li]set transparent target with blending OFF[/li][/ol]
Then the shader code is like this
out vec4 opaque_color_target;
out vec4 transparent_color_target;
if(coverage<SOME_THRESHOLD)
{
opaque_color_target=vec4(0.0, 0.0, 0.0, 0.0);
gl_DepthFragmentMask=false;
transparent_color_target=vec4(color, coverage);
}
else
{
/*
Note that we overwrite the transparent completely to nothing
*/
opaque_color_target=vec4(color, 1.0);
gl_DepthFragmentMask=true;
transparent_color_target=vec4(0.0, 0.0, 0.0, 0.0);
}
Once the buffers are drawn then one presents the buffers, essentially
RGB=opaque_color_target.rgb + transparent_color_target.a*transparent_color_target.rgb
With this thing we get coverage based anti-aliasing where edges do not overlap (edges overlapping is really not that common enough in terms of number of pixels for me to freak). My target is not handling “jaggies” of primitives, but rather expensive font shaders that draw glyphs as quads and I want to render opaque UI’s in an out-of-order fashion.
Currently, I emulate this with a two pass technique, but that means the font shader (or another expensive shader) gets run twice which, well, sucks. Also there is no guarantee that the front most “AA-crud-edge” will gets its day in the sun. On the subject of AA-based on primitives, there is this: http://www.khronos.org/registry/gles/extensions/NV/EGL_NV_coverage_sample.txt which kind of smells like the above, but it only handles the stuff at a primitive level, not a fragment level which is needed for font rendering and for that matter any fragment shader that uses discard.
One can naturally extend this to the idea of multiple edges on a fragment supported in much the same way one does out of order transparency (which in fact this reduces to)… but the main use case is to just avoid running a primitive and all its fragment stuff again to get around something that might be just an API oversight.