View Full Version : Stencil buffer

04-02-2009, 02:46 AM
Hi all

I have rendered a scene with the stencil test enabled. Now I want to render in white color all the places where the value of the stencil test is 2 or more (for instance).

What I do is to render a GL_QUAD for all the screen with the stencil test activated and just paint those pixels who passes the stencil test. That works fine.

I define the stencil function as this:

glStencilFunc(GL_EQUAL, popularPlaceValue-1, popularPlaceValue-1);

Where popularPlaceValue is an uint of my class. That renders in white color all the pixels where the stencil value is exactly the value popularPlaceValue.

What I want is to render in white all the pixels where stencil value <= popularPlaceValue.

Does anyone can help me?

04-02-2009, 02:59 AM
2 potential errors here :
- the third parameter is an AND bitmask, if you don't understand, keep it to all bit sets to 1 : 0xFFFFFFFF.
- EQUAL with (popularPlaceValue-1) as ref value should never show 'popularPlaceValue', but rather (popularPlaceValue-1).

From the doc it does not sound so complex :
glStencilFunc(GL_LEQUAL, popularPlaceValue, 0xFFFFFFFF);

04-02-2009, 03:34 AM
That was I tried at the beginning but I don't know why it doesn't work. Maybe the values of the stencil are wrong.

Thank you for your reply.

04-02-2009, 05:14 AM
It's most likely that your comparisons are the wrong way round. I had that problem when I first used the stencil - took ages to figure out!

The stencil test will pass if your reference value is {less,equal,greater} than the value in the stencil buffer.
You are probably assuming that the stencil buffer is compared to your ref value - it's not and the comparision works the otherway round.

glStencilFunc (GL_EQUAL, myRefValue, $FF); //pass stencil test if the ref value (equal) to stencil buffer

glStencilFunc (GL_LEQUAL, myRefValue, $FF); //pass stencil test if the ref value (Less or equal) to stencil buffer

04-02-2009, 06:07 AM
That's exactly what's happening. I was assuming the comparision in the wrong way. Really amazing error because I read the glStencilFunc documentacion several times and I didn't realize that.

Thank you for your reply.

04-02-2009, 09:39 AM
I read the spec over and over again as well....still could not figure it out. At one point I thought it was a driver bug. The only thing which stopped me reporting it was that nVidia are good with drivers and it's an unlikely bug to have persisted around for so long. I had access to other h/w with much older drivers and the same was happening there - therefore it had to be me (which it was).

Glad to help.