Antialiasing: polygon smoothing or global antialia

Hello.

I am working on scene antialiasing and i want to choose a best strategy for me.
Can you please help me… I cannot understand some things also.

My program uses shaders to draw meshes and apply light and material.

As far as i understood there are basically 2 strategies:

1. Use polygon smoothing.
   I really do not understand why and what it smoothes…
   For example (just hypothetically speaking), I have an object X (close to camera, Ad = 1, destination) and object Y (far from camera and is overlapped by object X, As = 1, source). Both objects are composed of several triangles.
   I switch of depth buffer, draw object X first, and then Y.

When I draw object Y, there are two possibilities:

a) As = Ad = 1 (no overlapping edges)
Resulting color is: Color = Ad*ColorX + min(As, 1-Ad)ColorY = 1ColorX + min(1, 0)*ColorY = ColorX.
This is correct since we keep the color of close ofbect X.

b) As = 1, Ad = 0.7 (any value) (object X has an edge, let’s assume this edge connects two triangles from object X, that is the edge is “inside” object X)
Resulting color is: Color = AdColorX + min(As, 1-Ad)ColorY = 0.7ColorX + min(1, 0.3)ColorY = 0.7ColorX +0.3ColorY.
This is not correct since edge is “inside” of X and we should not see any part of object Y.

The correct answer should be Color = ColorX.
c) As = 1, Ad = 0.7 (any value) (object X has an edge, let’s assume this edge is on the border of object X)
then we get same result as in b) but it is correct now.

So, I am sure I misunderstand something, but point b) is very confusing. Can you point me my mistake?

2. Second antialiasing method is a global antialiasing:

a) I understand Accumulation Buffer, but i think that it is better to move objects (each with different translation) instead of frustum. Thus, we will have same jitter independent on distance from camera, no over/under-blurred regions.
What is your opinion about object jitter instead of frustum jitter?

b) Use hardware technique described in NeHe . The question is: will it work for me since I already use my own written shaders?

Yes, multisampling (aka MSAA, aka multisample antialiasing) is completely compatible with writing your own shaders, and in general “just works” when you enable it. This is probably the simplest, most straightforward way to go. Just chose how many samples you want per pixel and then render your scene.

POLYGON_SMOOTH is very old and I probably predates fast MSAA. May require BLENDing too, which is a perf hit.

Multipass jittered blending is possible but overkill given fast MSAA which is ubiquitous.

1. Polygon Smoothing

Most recent graphics chips do not appear to support polygon smoothing at all, so this approach isn’t practical if you want to run on lots of different (and/or newer) graphics chips.

With that said, from your math, it looks like you are using glBlendFunc(GL_SRC_ALPHA_SATURATE, GL_DST_ALPHA). If you want to hide internal edges when using polygon anti-aliasing, the Red Book recommends glBlendFunc(GL_SRC_ALPHA_SATURATE, GL_ONE).

You need to render front-to-back, and avoid inter-penetrating triangles, but it works great for building smooth-edged opaque objects out of many triangles.

It represents “alpha-means-coverage” pretty well, it doesn’t produce correct results when “alpha-means-transparency”. (You should still be able to “see though” two 50%-transparent objects stacked together).

2a)
I was under the impression that the frustum jitter effectively offset the entire scene by some fraction of a pixel. This is pretty close to taking multiple samples per pixel (which is pretty close to rendering at a higher resolution). I don’t see how this would “over-blur” or “under-blur” objects, regardless of their distance. The frustum jitter isn’t quite as good as taking multiple samples per pixel because if the object is too small (< 1/2 pixel), it will get dropped in all jittered frames, whereas multiple samples might catch it with some samples. If all your primitives are >1 pixel in size, though, this shouldn’t be a problem.

The accumulation buffer will also enhance anti-aliasing within objects, not just along edges.

2b)

Yes, multi-sample anti-aliasing (MSAA) is definitely compatible with custom shaders.

Expect MSAA to be much faster than the accumulation buffer – but it won’t improve quality anywhere except along edges.

If the MSAA doesn’t give you enough AA, you could render with MSAA to a larger buffer (e.g. an offscreen buffer), and downsample. Or you could combine MSAA with the accumulation buffer.

Dark Photon and Kelvin, thanks for answer.
Ok. I think that MSAA suits me the most. I will implement it and see how it works)