I wrote a deferred renderer (deferred shading), with which I first attempted to use light stencil volumes in order to restrict the drawing region. This is done using a stencil buffer attached to the GBuffer FBO, using the format GL_DEPTH24_STENCIL8.
However, using stenciling as opposed to just rendering the light volume with depth testing (which still overdraws in many cases) cuts the frame rate in half instead of increasing it. Is the GL_DEPTH24_STENCIL8 format particularly slow, or is such performance normal for stenciling? Stenciling only half of the light volume and then using depth testing (no overdraw) was about as slow as stenciling everything. I tried not clearing the stencil buffer and just setting the stencil op to reset stenciled fragments, but still no dice. I also made sure that it does not write to the stencil buffer when testing against it, and I unbound the lighting shader when rendering the light volume. I also tried grouping light types in order to prevent shaders from being bound/unbound too often, but that didn't affect the frame rate.
This is pretty vague, but what constitutes "good" performance in a deferred renderer? I am trying to use light maps ATM, since I only get 70 fps in a scene with ~150 lights without any other effects on a Radeon HD6700M. The lights are frustum culled using an octree. With SSAO and edge detection AA, it goes to ~40 at times, which is unacceptable. Scene geometry is not a problem, without lighting it runs at > 450 fps. The lights fill entire rooms in the scene (not good for deferred, but it is usually only 1 light per room).
This leads to another question. How can I best integrate light maps in a deferred renderer? I tried using an emissivity buffer in the GBuffer which the light shaders use to reject fragments that are fully emissive, but this results in a high memory usage and was just as slow as rendering all lights dynamically. It still has to touch all of the fragments a light may effect emissive or not with this technique. I also tried using stenciling to stencil out emissive geometry to keep the lighting shaders from rendering to them, but I abandoned this when simply writing to the stencil buffer when rendering the level geometry already brought the frame rate to ~40fps.
Thanks for any help you can provide!