Stencils, Textures & Depth, a bug on ATI?

Dear all,

I’m developing a lighting simulation software and I have run into a very troublesome problem. Basically, I’m simulating lighting beams in a 3D environment. The effect I’m trying to achieve is:

  1. http://www.lewlight.com/files/SS_1.jpg
    Here I’m drawing the hit on the floor as a “cake with triangle pieces cut from the center”. As you can see in the wireframe outlines, these triangles pass through solids on the arch on stage as well as the stage steps on the left.

  2. http://www.lewlight.com/files/SS_1_2.jpg
    The previous image (1.) I draw the stage hit without depth testing, and this is what happens when you look at it from another angle. Hence, disabling depth testing is not a good idea.

  3. http://www.lewlight.com/files/SS_2.jpg
    This is image 1. with depth testing enabled. My idea is to blend image 1 and image 3 in the regions where the stage hit cake piece triangles intersect with 3D solids.

  4. http://www.lewlight.com/files/SS_3.jpg
    I have now created a stencil and painted the screen in red over the stencil to show you that I have successfully masked out the regions to blend (had I rotated to look at this like in SS_1_2, the stencil would not appear behind the wall, which is obviousley good.)

  5. http://www.lewlight.com/files/SS_4.jpg
    So. First I paint SS_2.jpg and over the stencil shown in SS_3.jpg I draw the image of SS_1.jpg (or, actually just the light of course).
    NOW YOU CAN SEE SOME ARTIFACTS! What the h*ck is that???

  6. http://www.lewlight.com/files/SS_5.jpg
    I have re-drawn the red over what is left of the stencil since step 5 (I decrease the stencil mask in steg 5 to see what’s left). You can clearly see that the red stencil mask portions left are those that were never drawn in step 5.

So, my question is… how come, when I disable the depth test and paint over the stencil, my ATI AX600 is not capable of filling the entire region? Am I drunk, or is the GPU drunk?

I’m not using culling, I’m using alpha blending of the vertices as well as in a 1-dimensional texture applied to smoothen the beam edges, but no other fancy stuff at all.

Any help would be greatly appreciated!

Regards,
Lars

Allright… I have had this verified on two NVidia GPU’s and I get the same result, so the GPU is most likely not drunk.

However, I don’t see how the stencil test can produce these artifacts? I have also verified that it’s no the blending, nor the texturemapping that causes the problem. So this has to be really basic. I have also tried the far/near, thinking there could be some problems with the depth buffer, but that seems not to be the problem either.

Is there some catch with the stencil buffer & the depth buffer that I don’t understand? If I could, I would post the code. But, basically, if I just paint my scene without depth testing, it’s all OK. But if I do the same with a glStencilFunc(GL_EQUAL, 1, ~0), I get these artifacts, no matter what I do with the depth buffer. :¨(

This is nuts.

Cheers,
Lars