Masking

How would I perform masking if my mask texture is separate from my color texture? In other words I have one texture with just alphas and another texture with just RGBs. For feasibility reasons it would not be practical for me to combine the mask and color bits into a single RGBA texture. … and I’ve got lighting going too.

The alpha stuff in the OpenGL guide always assumes the alphas are rolled in with the RGBs. Same with the blending and implementation of GL_DST_ALPHA in glBlendFunc is not universally supported.

The stencil buffer seems to be the solution. Reducing the mask from 8bits to 1bit is still acceptable. My current stencil testing code causes the shape of the quad to be written to the stencil buffer when I’m really just looking for the white areas of my mask.

Using version 1.1 and no extensions.

Thanks

I’ve never done this, but I think you can texture w/ alpha values only. Combining this with multipass texturing and alpha test might do the job.

When you draw the alpha texture into the stencil buffer, if you have alpha testing enabled and have the alpha test appropriately set, then it is possible just for the nonzero portion of the mask to be drawn. Then you can draw with the rgb texture using a stencil test. Instead of using a stencil buffer, you may use an alpha buffer. If you use an alpha buffer, then you will not need to use any alpha test. Either way, be sure you are using a 32 bpp mode.

Originally posted by Moz:
I’ve never done this, but I think you can texture w/ alpha values only. Combining this with multipass texturing and alpha test might do the job.

Hi Moz. I’ve gone and tested your suggestion first by texturing the alpha channel. No go. But here’s what happened.

I assume you ment to draw the alpha values first followed by the color texture. By alpha test I assume you ment glBlendFunc and not glAlphaFunc because glAlphaFunc only tests the alphas of incoming textures against a reference value and not the alphas in the framebuffer against a ref value. (Could I have been less wordy?)

Anyways it seems the reason why whats below didn’t work is because of line [8]. The glColor4f call of my quad sets the alpha in the framebuffer and not my mask. By varying the fourth parameter all, none or some of my color tex would appear. I thought of removing line [8] but then OpenGL would just use the alpha from what ever glColor call I did last. Bummer.

I hope I interpreted your suggestion correctly.

// loaded mask as a one component with internal GL_ALPHA format
glBindTexture( GL_TEXTURE_2D, maskName )
glTexImage2D( GL_TEXTURE_2D, 0, 1, width, height, 0, GL_ALPHA,
GL_UNSIGNED_BYTE, pMaskBits )

// loaded color bmp as 3 component with internal format GL_BGR_EXT
glBindTexture( GL_TEXTURE_2D, colorName )
glTexImage2D( GL_TEXTURE_2D, 0, 3, _width, _height, 0, GL_BGR_EXT,
GL_UNSIGNED_BYTE, pColorBits )

// create plane for textures to sit on
glNewList( listName, GL_COMPILE )
glBegin( GL_QUADS )
glNormal3f( 0.0f, 0.0f, 1.0f )
[8] glColor4f( 0.5f, 0.0f, 0.0f, 0.5f )
glTexCoord2fv( bottomRight ); glVertex2i( 40, -40 );
glTexCoord2fv( topRight ); glVertex2i( 40, 40 );
glTexCoord2fv( topLeft ); glVertex2i( -40, 40 );
glTexCoord2fv( bottomLeft ); glVertex2i( -40, -40 );
glEnd( )
glEndList( )

glClear( GL_COLOR_BUFFER_BIT );

// no depth testing so mask does not fight with color tex
glDisable( GL_DEPTH_TEST );

// write to alpha layer only
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE );

// Draw mask
glEnable( GL_TEXTURE_2D );
glTexEnvf( GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE );
glBindTexture( GL_TEXTURE_2D, maskName );
glCallList( listName );

glEnable( GL_LIGHTING );
glEnable( GL_LIGHT0 );
glLightfv( GL_LIGHT0, GL_POSITION, lightPos );

// Write to RGB layers. Protect alphas.
glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_FALSE );

// Set to modulate because we got lights going for color tex.
glTexEnvf( GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE );

// Draw color texture
glEnable( GL_BLEND );
glBlendFunc( GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA );
glBindTexture( GL_TEXTURE_2D, colorName );
glCallList( listName );
glDisable( GL_BLEND );

glDisable( GL_LIGHT0 );
glDisable( GL_LIGHTING );
glDisable( GL_TEXTURE_2D );

I’m off to try DFrey’s suggestions.

Originally posted by DFrey:
When you draw the alpha texture into the stencil buffer, if you have alpha testing enabled and have the alpha test appropriately set, then it is possible just for the nonzero portion of the mask to be drawn. Then you can draw with the rgb texture using a stencil test

Hi DFrey. I’ve tried your suggestion about using stencil and alpha testing prior to drawing the color texture. Unfortunately that didn’t seem to work either.

According to the guide if a fragment passes the alpha test then the fragment moves onto the stencil test. However if the fragment fails the alpha test then testing stops and OpenGL movies onto the next fragment. Fine.

This is all very nice had OpenGL actually evaluated the alphas in my mask. Instead OpenGL chose to evaluate the alpha from glColor4f (line [8]) that was used to draw the quad my texture sits on. By changing glColor4f’s alpha I saw everything or nothing. Removing the glColor4f call wouldn’t have changed anything because OpenGL would then use what ever alpha was set from the most recent glColor call.

Masking with the alpha mask split from the color texture is proving to be a royal pain.

// loaded mask as a one component with internal GL_ALPHA format
glBindTexture( GL_TEXTURE_2D, maskName )
glTexImage2D( GL_TEXTURE_2D, 0, 1, width, height, 0, GL_ALPHA,
GL_UNSIGNED_BYTE, pMaskBits )

// loaded color bmp as 3 component with internal format GL_BGR_EXT
glBindTexture( GL_TEXTURE_2D, colorName )
glTexImage2D( GL_TEXTURE_2D, 0, 3, _width, _height, 0, GL_BGR_EXT,
GL_UNSIGNED_BYTE, pColorBits )

// create plane for textures to sit on
glNewList( listName, GL_COMPILE )
glBegin( GL_QUADS )
glNormal3f( 0.0f, 0.0f, 1.0f )
[8] glColor4f( 0.5f, 0.0f, 0.0f, 0.0f )
glTexCoord2fv( bottomRight ); glVertex2i( 40, -40 );
glTexCoord2fv( topRight ); glVertex2i( 40, 40 );
glTexCoord2fv( topLeft ); glVertex2i( -40, 40 );
glTexCoord2fv( bottomLeft ); glVertex2i( -40, -40 );
glEnd( )
glEndList( )

glClearStencil( 0 );
glClear( GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT );

glDisable( GL_BLEND );

// no depth testing so mask does not fight with color tex
glDisable( GL_DEPTH_TEST );

// Leave the color buffer alone
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );

// any alpha other than 0 moves onto the stencil test
glEnable( GL_ALPHA_TEST );
glAlphaFunc( GL_GREATER, 0 );

// any non-zero alpha becomes white in stencil buffer
// alphas==zero don’t make it this far and are discarded
// before getting to the stencil test.
glEnable( GL_STENCIL_TEST );
glStencilMask( 1 );
glStencilFunc( GL_ALWAYS, 1, 1 );
glStencilOp( GL_REPLACE, GL_REPLACE, GL_REPLACE );

// Draw mask
glEnable( GL_TEXTURE_2D );
glTexEnvf( GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE );
glBindTexture( GL_TEXTURE_2D, maskName );
glCallList( listName );
glDisable( GL_ALPHA_TEST );

// Update color buffer where stencil is white and
// deny updates to stencil buffer.
glStencilFunc( GL_EQUAL, 1, 1 );
glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
glEnable( GL_LIGHTING );
glEnable( GL_LIGHT0 );
glLightfv( GL_LIGHT0, GL_POSITION, lightPos );

// Draw color texture
glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_FALSE );
glTexEnvf( GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE );
glBindTexture( GL_TEXTURE_2D, colorName );
glCallList( listName );

glDisable( GL_LIGHT0 );
glDisable( GL_LIGHTING );
glDisable( GL_STENCIL_TEST );
glDisable( GL_TEXTURE_2D );

Well, that code looks like it should work. Indeed it looks very similar to some of my code that I know works. The only difference is you use a GL_ALPHA texture whereas I used a GL_RGBA texture. That was the only way I could get it to work if I didn’t also enable a destination alpha buffer. If I did have an alpha buffer enabled, then I could get GL_LUMINANCE_ALPHA type textures to work even though I didn’t use the destination alpha, but GL_ALPHA textures still failed.
Starting to make me think that NVIDIA OpenGL implementation is not very robust when it comes to GL_ALPHA textures. Maybe Matt can clarify things.

Maybe Matt can clarify things.[/b]

Who’s Matt?

I saw an odd thing at http://msdn.microsoft.com/library/default.asp
You can find it by following these subsection titles in the table of contents:
Platform SDK
Graphics and Multimedia
OpenGL
Win32 Extensions to OpenGL
OpenGL Reference
GL functions
glTexImage2D

The description for parameter [format] when using GL_ALPHA is:
“Each element is a single red component.”
and general comments
“A texture image can have up to four components per texture element, depending on components. A one-component texture image uses only the red component of the RGBA color extracted from pixels.”

Intuitively I would have thought a one component GL_ALPHA texture uses the alpha component. Assuming this stuff about the red component is true that would mean I would have to:
draw my mask to the red layer
copy the red layer to the alpha layer
clear the red layer
finally draw my color texture?
That’s stupid.

Matt is an NVIDIA OpenGL developer that frequents these forums. And I believe that MSDN article is just a copy and paste error.

[This message has been edited by DFrey (edited 12-09-2000).]