Trouble with the stencil buffer

I’ve got a bunch of triangles, of 3 different types. I want to render the first two types of triangles, and then render the third type----but I only want fragments to be drawn for the third type on pixels covered by triangles of both of the first two types. Triangles of a single type do not overlap, but may share a side.

I’ve set up an FBO with a GL_DEPTH24_STENCIL8_EXT depth renderbuffer attached to GL_DEPTH_ATTACHMENT_EXT and a suitable color buffer attached to GL_COLOR_ATTACHMENT0_EXT.

Now I’m doing this:


    glEnable(GL_STENCIL_TEST);
    glClear(GL_STENCIL_BUFFER_BIT);
    glStencilOp(GL_INCR,GL_INCR,GL_INCR);
    glStencilFunc(GL_ALWAYS,0,-1);

    glBegin(GL_TRIANGLES);
    for (unsigned i = 0; i < tris.size(); ++i)
    {
        if (tris[i].getType() == 0 ||
            tris[i].getType() == 1)
            drawTriangle(tris[i]);// wraps glVertex2d calls
    }
    glEnd();

    glStencilOp(GL_KEEP,GL_KEEP,GL_KEEP);
    glStencilFunc(GL_LESS,1,-1);// Only if at least two triangles have been drawn to this pixel.

    glBegin(GL_TRIANGLES);
    for (unsigned i = 0; i < tris.size(); ++i)
    {
        if (tris[i].getType() 2)
            drawTriangle(tris[i]);
    }
    glEnd();
  
    glDisable(GL_STENCIL_TEST);
    glDisable(GL_BLEND);
    glEnable(GL_DEPTH_TEST);
    
    assert(!throwOnOpenGLError());
    vector<int> ints(900*900);
    glReadPixels(0,0,900,900,GL_STENCIL_INDEX,GL_INT,&ints[0]);

    assert(!throwOnOpenGLError());

However, visually the stencil clipping does not appear to take place, and the glReadPixels is generating GL_INVALID_OPERATION, which could mean that there is no recognized stencil buffer. Does anyone have any ideas what might be wrong?

Where do you call glReadPixels ? Are you sure your fbo supports stencil attachment ? Do you check opengl erros before ? (mainly during fbo stencil creation ?)

There are no OpenGL errors prior to the glReadPixels().

Am I sure the FBO has a packed depth/stencil attachment? Fairly certain, but not 100%. Is there a glGet() I can use to check? My FBO setup code is fairly encapsulated so it’s not as simple as pasting a block of OpenGL calls.

I tried this instead:
glReadPixels(0,0,900,900,GL_DEPTH_STENCIL_EXT,GL_UNSIGNED_INT_24_8_EXT,&ints[0]);

But I’m still getting the invalid operation error.

I’ve set up an FBO with a GL_DEPTH24_STENCIL8_EXT depth renderbuffer attached to GL_DEPTH_ATTACHMENT_EXT

Well there’s your problem. You only attached the depth buffer part to the FBO. You need to attach both parts. Your renderbuffer may contain both depth and stencil information, but unless you attach the renderbuffer to both the depth and stencil attachments, that doesn’t mean the FBO itself has both attachments.

Attach your depth/stencil buffer to the GL_DEPTH_STENCIL_ATTACHMMENT.

Thanks, that did it! At least the glReadPixels is now working. I’m still getting some results which I don’t understand on the stenciling itself, but I’ll ask about that tomorrow if I haven’t worked it out by then.