I have read all about GL_POLYGON_SMOOTH, and it seems simple. I am rendering some simple projected shadows to an FBO, so I thought this type of antialiasing would be perfect.
@ZbuffeR: I think the shadow in the image from the link you posted is just a normal planar shadow which does NOT use a texture at all! GL_POLYGON_SMOOTH won’t make the shadow image in the texture soft!
@Trenki: my understanding of OP technique is to render shadow caster object with polygon smooth to a an FBO (instead of rendering to main framebuffer using stencil), and then using the FBO as darken texture.
@Leadwerks : speaking about basic tricks, maybe you can simply replace polygon smooth by multisampling if supported, or render to larger FBO and smooth it yourself. Whether these hacks are better than proper PCF is another question …
Multisampling in an FBO is not supported on all cards (e.g. ATI)! And Multisampling alone won’t make the shadows less blocky. For that you would have to increase the size of the rendered to texture.
Then added your settings for the part rendered to FBO.
It did not work with glBlendFunc GL_SRC_ALPHA_SATURATE,GL_ONE, dunno why.
A simple glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA); did the trick instead.
An RGBA FBO is not needed, RGB is enough (only src component sampled).
Tested on Ubuntu Edgy with quite oldish drivers:
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce 6800 LE/AGP/SSE2/3DNOW!
OpenGL version string: 2.0.2 NVIDIA 87.76