I’m in the middle of porting some HLSL shaders to opengl (with CG) and I’m struggling a bit with the correct opengl stencil statements…
I have to states that need to be set for the stencil buffer :
The last 2 arguments of StencilOp treat about when stencil test passed, the first of them when depth buffer test failed, the second one when depth buffer test passed. So, I think you should keep the same values for the last 2 arguments, regarding what you provided.
glClearStencil doesn’t actually clear the stencil buffer, it just sets the clear value. You need to to call glClear with GL_STENCIL_BUFFER_BIT included to clear it.
//first one
glClearStencil(0);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
Are you trying to initially set the stencil to 1 for every pixel of the depth/stencil buffer?
In which case all you need is:
glClearStencil (1);
glclear (GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
and with this:
//second one
glClearStencil(0);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
You are trying to find areas of the stencil buffer which have been tagged with a value of 1 (which initially they all are)?
In which case, you just need:
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);