Thank you for your help. Please see my detailed explanation as follows:
What I basically want to do is to chop a small triangle
area off a big triangle. I use stencil buffer to do this.
The following is the code with the comments.
glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
//init stencil buffer
glEnable(GL_STENCIL_TEST);
glClearStencil(0x0);
glClear(GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
//turn off color buffer so that the small triangle
//to draw to set the stencil buffer will not show up
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
//draw the red smaller triangle to set the stencil buffer
glColor3f(1.0, 0, 0);
glBegin(GL_TRIANGLES);
//the big triangle is at depth -22
glVertex3f(-1.0, 0, -22);
glVertex3f(1.0, 0, -22);
glVertex3f(0, 1, -22);
glEnd();
//draw the blue big triangle, of which the part overlaps with the
//smaller triangle is chopped off
glColorMask(GL_TRUE,GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_NOTEQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glBegin(GL_TRIANGLES);
glColor3f(0.0, 0, 1.0);
//the big triangle is at depth -21
glVertex3f(-2.0, 0, -21);
glVertex3f(2.0, 0, -21);
glVertex3f(0, 2, -21);
glEnd();
glDisable(GL_STENCIL_TEST);
The code works perfectly fine on ATI graphics cards; however, on
nVidia cards, it does not work, nothing is chopped off the big
triangle. However, if I change the z-value of the big triangle from
-21 to -23, which makes the big triangle behind the smaller one, it
works on nVidia cards. In both cases, ATI cards work correctly.
Therefore, this makes me to belive for nVidia, the stencil buffer
is somehow associated with depth buffer. Nonetheless, for standard
opengl, stencil buffer is simply a 2D mask.
Any help is highly appreciated.