Stencil test problem

[anuchka]

Hi

1.I'm using stencil test to draw ceilings for non-convex polygons
Sometimes I'm getting strange display ,like the attached image ,i.e. I can see transparent lines ,and also white stripes around the frame

http://img238.imageshack.us/my.php?image=screenhunter023eu1.jpg

My code:

glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_INVERT, GL_INVERT);
for (int ib = 0; ib < iNumBuildings;ib++)
{
glBegin(GL_TRIANGLE_FAN);
for(int iw = 0;iw < buildings[ib].iNumWalls;iw++)
glVertex4f(.vertex coordinates..);
glEnd();
}

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

for (int ib = 0; ib < iNumBuildings;ib++)
{
glBegin(GL_TRIANGLE_FAN);
for(int iw = 0;iw < buildings[ib].iNumWalls;iw++)
glVertex4f(...vertex coordinates...)
glEnd();
}
glDisable(GL_STENCIL_TEST);


Any suggestions?

Thanks

[trinitrotoluene]

After some simple testing, I found these stencil state setup.



//Draw the geometry only in the stencil buffer
//to avoid writing some unwanted value in the depth buffer.
glEnable(GL_STENCIL_TEST);
glDisable(GL_DEPTH_TEST);
glColorMask(GL_FALSE,GL_FALSE,GL_FALSE,GL_FALSE);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_INVERT);

//Draw concave geometry


//Draw the geometry only where the mask is 1 with depth test //enable
glEnable(GL_DEPTH_TEST);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 0x1, 0x1);
//Reset the stencil mask to 0 where it was 1. Useful if we want //to draw many concave geometry
glStencilOp(GL_KEEP,GL_INVERT,GL_INVERT);
//Draw concave geometry again
glDisable(GL_STENCIL_TEST);


Hope this code help and is bug free.

[anuchka]

Both versions (your and mine) do the same,but thanks

Problem solved ,it was because of
glEnable(GL_POLYGON_SMOOTH);

I think also enabling/disabling depth buffer doesn't matter here....( or does ?)

[trinitrotoluene]

I am glad to know that the rendering of your concave polygon now work.

With the code you post, I have to modify the the depth func (GL_LEQUAL) to be able to see the concave poly. But if a draw some geometry behind, the triangle(s) part who are not inside the concave poly hid and I see the background color. It is for this
problem I have to disable the depth buffer when writing the stencil buffer the first time.