Stencil works with GL_FLOAT but not with GL_BYTE

I’m trying to get some stencil operations to work. I’m displaying a 1024x1024 pixels image to the screen, but I would like to not show some parts.

I tried to set up a stencil with GLbytes like this:

GLbyte m_stencildata = new GLbyte[1024*1024];
GLbyte *p=m_stencildata;
for ( int y=0; y<1024; y++ )
    for ( int x=0; x<1024; x++, p++ )
        *p=((y % 20)>10)?0:0x7F;

glEnable(GL_STENCIL_TEST);
glPixelTransferf(GL_MAP_STENCIL, GL_FALSE);
glClear(GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_ALWAYS, 1, 1);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glRasterPos2i(0, 0);
glDrawPixels(1024,1024,GL_STENCIL_INDEX,GL_BYTE,m_stencildata);
glStencilFunc(GL_EQUAL, 1, 1);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

delete [] m_stencildata;

Unfortunately, this does not work. If I do the same with GL_FLOAT:

GLfloat m_stencildata = new GLfloat[1024*1024];
GLfloat *p=m_stencildata;
for ( int y=0; y<1024; y++ )
    for ( int x=0; x<1024; x++, p++ )
        *p=((y % 20)>10)?0.0:1.0;
[...]
glDrawPixels(1024,1024,GL_STENCIL_INDEX,GL_FLOAT,m_stencildata);
[...]

everything works as expected. However, I expect this to be slower than the GL_BYTE approach. I appreciate any pointers on how to get this to work with GL_BYTE!

Koen

Try using GL_UNSIGNED_BYTE instead. Stencil is unsigned anyway, so your two code snippets are not equivalent. The first writes 0x7F, the latter 0xFF.

OK, I changed the code to:

GLubyte m_stencildata = new GLubyte[1024*1024];
GLubyte *p=m_stencildata;
for ( int y=0; y<1024; y++ )
    for ( int x=0; x<1024; x++, p++ )
        *p=((y % 20)>10)?0:0xFF;
[...]
glDrawPixels(1024,1024,GL_STENCIL_INDEX,GL_UNSIGNED_BYTE,m_stencildata);
[...]

but it doesn’t work either.

As a sidenote: I tried GL_BITMAP as well, which worked quite well. I guess that rules out problems in other parts of the code.

I also tried GL_UNSIGNED_SHORT, with no luck.