Transparency Antialiasing/CSAA with Depth Stencil

First of all: appologies for a long post - but I wanted to include as much detail as possible to avoid needless question/answer posts if i’d just thought about what I was writing in the first place.

I’m trying to add some sort of anti aliasing for transparent objects to my deferred engine. At the moment the engine has a separate forward pass phase to blend transparent objects like grass. Another possible rendering path is to render the grass objects as part of the GBuffer phase with Alpha Test enabled.

Having read some nVidia documents on AA, CSAA looks like a nice and easy way to introduce multisampling and will propably serve me well for the grass objects. Now I wish to add anti aliasing to this part of the engine - the grass - using MSAA or CSAA or Transparency Anti Aliasing.

I’m aware that GL 3.2 now supports multisample textures - but to use this I’d have to rewrite my existing shader library and I’d prefer not to do that as I have a mostly legacy GL 2.1 engine.

What I have done is create a wrapper around the Gbuffer phase of the engine - and switch to using Multisample renderbuffers instead of the current implementation’s MRT render textures.
To do this I created an additional MRT GBuffer containing 3 multisample RenderBuffers (not textures) and used this new framebuffer when rendering the GBuffer phase. I then added an FBO BLIT at the end to ‘resolve’ each attachment into the original MRT GBuffer - so as far as the rest of the application is concerned there is no multisampling, just the usual MRT I have always had containing a Depth/Stencil texture and Colour,Normal and Position texture attachments. This is nice as there are a lot of dependancies upon the depth buffer and colour texture attachments (for example post processing).

Although I have sucessfully created the MRT renderbuffers for Multisample - it did not create a depth WITH stencil render buffer (combined). I did not get any GL errors or even a framebuffer incomplete, but a query to return the number of stencil bits gave 0. The stencil is a cruital element of the framebuffer for my engine, as the lighting depends upon it- and this is the bit giving me the most problems.

My Engine is tested on GForce 8 and Radeon 4850. Both can not give me a depth/stencil renderbuffer, yet alone a multisampled depth/stencil buffer.

Here’s my code for the ARB Framebuffer - create a depth Renderbuffer
glGenRenderbuffers( 1, @FrameBuffers[bufferID].FBDepth ); // allocate shared Depth render target.
glBindRenderbuffer( GL_RENDERBUFFER, FrameBuffers[bufferID].FBDepth );
if samples > 0 then
begin
if (CoverageSamples >0 ) AND (NVMultiSampleCoverage) then glRenderBufferStorageMultiSampleCoverageNV (GL_RENDERBUFFER,CoverageSamples, samples,depthBuffer.InternalFormat,width,height)
else glRenderBufferStorageMultiSample (GL_RENDERBUFFER, samples,depthBuffer.InternalFormat, width,height);
end else glRenderbufferStorage( GL_RENDERBUFFER, depthBuffer.InternalFormat, Width , Height );
end;

//Tested on both ATI and nVidia - use the DepthBuffer ID and attach to FBO. Do not use: GL_DEPTH_STENCIL_ATTACHMENT
glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_DEPTH_ATTACHMENT,GL_RENDERBUFFER, FrameBuffers[bufferID].FBDepth);

Here’s the Stencil Renderbuffer creation - i.e. resuse the depth buffer as the stencil
glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_STENCIL_ATTACHMENT,GL_RENDERBUFFER, FrameBuffers[bufferID].FBdepth);
glGetRenderbufferParameteriv (GL_RENDERBUFFER,GL_RENDERBUFFER_STENCIL_SIZE, @depthBuffer.GLAllocatedStencilSize);

My Depth format is Depth24_Stencil8, so I expect Depth and Stencil to share the same renderbuffer. As I said, creating a separate stencil render buffer with the code below give ‘Frame buffer unsupported’

glGenRenderbuffers( 1, @FrameBuffers[bufferID].FBstencil ); // allocate shared stencil render target.
glBindRenderbuffer(GL_RENDERBUFFER, FrameBuffers[bufferID].FBstencil);
glRenderbufferStorage(GL_RENDERBUFFER, GL_STENCIL_INDEX8, width , height);
if (fbDepthStencil in Template.flags) then glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_DEPTH_STENCIL_ATTACHMENT,GL_RENDERBUFFER, FrameBuffers[bufferID].FBstencil)
else glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_STENCIL_ATTACHMENT,GL_RENDERBUFFER, FrameBuffers[bufferID].FBstencil);
glGetRenderbufferParameteriv (GL_RENDERBUFFER,GL_RENDERBUFFER_STENCIL_SIZE, @depthBuffer.GLAllocatedStencilSize);

This is what i need help with:

  1. Creating an ARB Multisample Render buffer for Depth/stencil (D24_S8). Am I attaching/creating the stencil renderbuffer incorrectly?

  2. Coverage Sample Anti Aliasing (CSAA). Is this only available by using RenderbufferStorageMultisampleCoverageNV - ie not avaiable using the ARB framebuffer Multisample RenderbufferStorage.

  3. MSAA (ARB_MultiSampling and now part of GL 1.3+). If CSAA is being used, do we still need to glEnable (GL_MULTISAMPLE) or is this only for MSAA. Do I need this enable state at all?

  4. If CSAA is not part of the ‘core’ GL 3.0+, then why are there references to GL_SAMPLE_TO_COVERAGE,GL_SAMLE_VALUE and SampleCoverage mentioned in the spec on page 197, section 4.1 ‘PER-FRAGMENT OPERATIONS’ of the GL3.3_Core_20100311_withchanges.pdf ?

  5. How to actually ‘turn on’ CSAA if RenderbufferStorageMultisampleCoverageNV is used.
    Do we need to glEnable (GL_SAMPLE_TO_COVERAGE) and glEnable (GL_SAMPLE_VALUE)

  6. How is the value of SAMPLE_COVERAGE_VALUE as set by the function gl_SampleCoverage used to control the anti aliasing?

I guess what I’m after here guys as a little help clearing up how to implement MSAA, CSAA and/or Transparency Anti Aliasing. The examples in nVidia SDK 9.52 don’t really help - the best visual example is a Transparency AA demo - but this is for DX7 and the source does not help. Another good demo id the nVidia ‘Froggy AA’ demo. Although it enumerates every single FBO format imagineable (sRGB, 16 bit float/fixed, etc) it does not include any Depth/Stencil formats. I wonder why?

Anyone with a working example with key source snippets they can share?

You want GL_DEPTH24_STENCIL8. See this extension spec for some source code on how to use it: ARB_framebuffer_object

  1. Coverage Sample Anti Aliasing (CSAA). Is this only available by using RenderbufferStorageMultisampleCoverageNV - ie not avaiable using the ARB framebuffer Multisample RenderbufferStorage.

Yes, I think so.

  1. MSAA (ARB_MultiSampling and now part of GL 1.3+). If CSAA is being used, do we still need to glEnable (GL_MULTISAMPLE) or is this only for MSAA. Do I need this enable state at all?

CSAA is basically an optimization of MSAA, so yes I believe you should have to enable multisample rasterization (GL_MULTISAMPLE) to make use of it.

  1. If CSAA is not part of the ‘core’ GL 3.0+, then why are there references to GL_SAMPLE_TO_COVERAGE,GL_SAMLE_VALUE and SampleCoverage mentioned in the spec on page 197, section 4.1 ‘PER-FRAGMENT OPERATIONS’ of the GL3.3_Core_20100311_withchanges.pdf ?

NVidia didn’t invent the term “coverage” with CSAA. This term already had meaning for multisample rasterization, and NVidia just happened to use it in the name of their mode.

  1. How to actually ‘turn on’ CSAA if RenderbufferStorageMultisampleCoverageNV is used.
    Do we need to glEnable (GL_SAMPLE_TO_COVERAGE) and glEnable (GL_SAMPLE_VALUE)

Don’t know where you’re pulling those symbols from, but throw away the source! They don’t exist. And no, SAMPLE_ALPHA_TO_COVERAGE does not have anything to do with CSAA specifically. It pertains to MSAA (and thus CSAA as well).

I think you just create a CSAA render target, enable multisample rasterization, and roll. Though I confess not to have coded for it specifically.

For window CSAA targets, you can end up using CSAA indirectly because several of NVidia’s built-in FSAA modes are CSAA-based, and (AFAIK) all you have to do is enable multisample rasterization and draw.

  1. How is the value of SAMPLE_COVERAGE_VALUE as set by the function gl_SampleCoverage used to control the anti aliasing?

Again, this is just a basic MSAA thing (see ARB_multisample or the an OpenGL spec), which should apply to CSAA as well.

With SAMPLE_COVERAGE enabled, the value you pass glSampleCoverage is ANDed with the coverage value. What does this mean? The “coverage” value is essentially used as a multisample dither mask, so this allows you to “dial up and down” the sample density of that dither mask. In other words, it allows you to do sort of a “poor man’s transparancy” without any alpha blending merely by varying the percentage of subsamples that are written vs. those that are just thrown away (due to the dither mask). On the net by any other name, this is often called “screen door transparency”. Though more commonly alpha is used as the coverage value, so you’d use SAMPLE_ALPHA_TO_COVERAGE instead.

Anyway, I’d suggest you read up on MSAA, get up on what that actually means, and then target your rendering to an MSAA or CSAA mode.

Thankyou for your reply.
I must confess to being a little confused by the spec, so upon reflection I have mis-read that page and made up GL_SAMPLE_TO_COVERAGE assuming that’s what it was taking about. I know realise that the enable states are sample_to_coverage, sample_to_one and sample_coverage.

Thanks again for clearing up some of the MSAA/CSAA terminology.

However, I still need to know if people can sucessfully create a renderbuffer with compined DEPTH24_STENCIL8 as my code (as posted above) fails - the stencil is returned as allocated=0.

I can’t create separate stencil buffers either as this is ‘unsupported’.

Yes, either as a renderbuffer or a texture. This query works with either case when the FBO is bouind and returns 8:

  glGetFramebufferAttachmentParameteriv( GL_FRAMEBUFFER_EXT,
                                         GL_STENCIL_ATTACHMENT_EXT,
                                         GL_FRAMEBUFFER_ATTACHMENT_STENCIL_SIZE, 
                                         &stsize );

And this works in the case of a renderbuffer if the renderbuffer is bound and returns 8:

  glGetRenderbufferParameteriv ( GL_RENDERBUFFER,
                                 GL_RENDERBUFFER_STENCIL_SIZE, 
                                 &stsize );

Make sure you are using these parameters to create the depth/stencil attachment:


iformat = GL_DEPTH24_STENCIL8;
format  = GL_DEPTH_STENCIL;
type    = GL_UNSIGNED_INT_24_8;

Doesn’t this apply to depth_stencil as a texture? The type and format are used when creating the texture for the depth/stencil and not renderbuffers.

I specifically need a FBO with renderbuffers as I already have an FBO with depth_stencil texture. I need the renderbuffer because I can then use the mulitsample renderbuffers API - which does not exist for textures (unless GL 3.3 and I don’t want that).

With experimenting, I have found that I can not create renderbuffers with depth_stencil - hence I need to know what’s wrong with my code.

So, for example, I use the following right after creating the depth render buffer:

glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_DEPTH_ATTACHMENT,GL_RENDERBUFFER, FBDepth);
…BUT whould I be using GL_DEPTH_STENCIL_ATTACHMENT instead?
…Now what about the stencil…do I then use:
glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_STENCIL_ATTACHMENT,GL_RENDERBUFFER, FBdepth);

True, so just use the iformat. Ignore the format/type.

With experimenting, I have found that I can not create renderbuffers with depth_stencil - hence I need to know what’s wrong with my code.

Ok, time to post a short GLUT test program illustrating your problem so we can find the problem. Use this as a template.

So, for example, I use the following right after creating the depth render buffer:

glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_DEPTH_ATTACHMENT,GL_RENDERBUFFER, FBDepth);
…BUT whould I be using GL_DEPTH_STENCIL_ATTACHMENT instead?
…Now what about the stencil…do I then use:
glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_STENCIL_ATTACHMENT,GL_RENDERBUFFER, FBdepth);

That’s fine. If it’s a depth+stencil iformat, just set it on both attachment points.

Your card does support EXT_packed_depth_stencil, correct?

Ok got it all working. Framebuffer renderbuffer is now depth stencil enabled using:

glFramebufferRenderbuffer(GL_FRAMEBUFFER,GL_DEPTH_STENCIL_ATTACHMENT,GL_RENDERBUFFER, FrameBuffers[bufferID].FBDepth);

As for stencil, no extra renderbuffer or attachment is required. The number of stencil bits can be queried via getFramebufferattachmentparameter - so thanks that was a good suggestion!

The AA sample alpha to coverage works a treat with just two glEnables. It truely is a technical wonder! Say goodby to depth sorting !