Masking and semi-transparency together

I want to draw a figure like one in a game. It needs to be “masked” and also semi-transparent (see-though). I have tried so many things to get it to work without a halo of the interpolated transparency color around the figure. I know I can use NEAREST interpolation but I need quality. Here is my code:

byte blendingEna= glIsEnabled(GL_BLEND);
	//byte alphaTestEna= glIsEnabled(GL_ALPHA_TEST);
	byte texture2DEna= glIsEnabled(GL_TEXTURE_2D);
	if(!texture2DEna)
		glEnable(GL_TEXTURE_2D); //slows down fps but necessary
	if(drawAlpha)
	{
		float a= (float)alpha/255.0f;
		glColor4f(1.0f, 1.0f, 1.0f, a);
		glEnable(GL_BLEND);
		glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
	}
	if(drawMask)
	{
		//if(drawMask)
		{glDepthMask(0); glDisable(GL_DEPTH_TEST);
		glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_ENV, GL_MODULATE);
			//glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE);
			//glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE);
			//glEnable(GL_ALPHA_TEST);
			//glAlphaFunc(GL_GREATER, .9f); //leavs no transparency color but creats blockiness at edges but does not work with drawAlpha is below .9f.
			//glAlphaFunc(GL_NOTEQUAL, 0);
			if(!drawAlpha)
			{
			glColor4f(1,1,1,1); glEnable(GL_BLEND); //GL_MODULATE uses GL_BLEND
			glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA); //helps smooth edges
			//glEnable(GL_POLYGON_SMOOTH); glPolygonMode(GL_POLYGON_MODE, GL_POLYGON_SMOOTH);
			//glBlendFunc(GL_SRC_ALPHA_SATURATE, GL_ONE);
			}
		}
		//glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
		glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
	}
	else if(HighQualDrawing)
	{
		glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
	}
	else
	{
		glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
	}
	//glColorMask(1,1,1,0); //use it to turn off mask (if there is one) disables alpha channel.
	glBindTexture(GL_TEXTURE_2D, abmp.pTex->id);
	glBegin(GL_QUADS);
    //top left
	glTexCoord2f(abmp.pTex->coordX, abmp.pTex->coordY);	glVertex2i(loc.x, loc.y);
	//bottom left
	glTexCoord2f(abmp.pTex->coordX, abmp.pTex->coordB);	glVertex2i(loc.x, loc.y+abmp.Bounds.h);
	//bottom right
	glTexCoord2f(abmp.pTex->coordR, abmp.pTex->coordB);	glVertex2i(loc.x+loc.w, loc.y+loc.h);
	//top right
	glTexCoord2f(abmp.pTex->coordR, abmp.pTex->coordY);	glVertex2i(loc.x+loc.w, loc.y);
    glEnd();
	if(drawAlpha)
	{
		if(!blendingEna)
			glDisable(GL_BLEND);
	}
	if(drawMask)
	{
		if(!blendingEna)
			glDisable(GL_BLEND);
		//if(!alphaTestEna)
			//glDisable(GL_ALPHA_TEST);
	}
	if(!texture2DEna)
		glDisable(GL_TEXTURE_2D);
	return 1;

Normally I’d move this to the beginner’s section but given the history of this post I’ll leave it here. I don’t want to discourage you from posting.

You have a few options but there are issues with each.

The two main problems you must contend with are transparency interacting with the depth buffer and the texel interpolation of color texels with alpha texels.

You can chose to blend or not blend, you can choose to sort or not sort, you can set alphafunc and ref values to cull pixels (with or without blending).

Flood filling the transparent texels in a texture image with colors to match the opaque portion will fix texel interpolation but this may not be required depending on the source image.

The highest quality solution is to enable blending and sort.

You can also have blendingh off but cull alpha fragments at about 0.5, again with good content. This avoids the need for a sort, allows bilinear interpolation and can smooth the silhouette edge to avoid jagginess if your alpha channel is reasonably well filtered in the source image, but you don’t get soft semi-transparent edges around transparent stuff (you may preffer this). It can also solve various issues like multipass problems.

P.S. there is a more exotic option of using alpha to mask with a multisample buffer but it isn’t generally applicable and has quality issues of its own.

I have had a ton of opinions on this problem but none of them have worked so far. Everyone mentions sorting but I don’t think it is applicable to me because I’m in ortho mode. However, I would use the sort and blend method if I was in 3-D.

Please, if you wouldn’t mind, take the code above and post it back fixed up so that I can have:
linear interpolation, a mask color(alpha channel; 32bpp), and semi-transparency all at once without getting that outline around the figure.

Write your own code.

Sorting works in ortho.

Try applying the ideas given to you and I’m sure you’ll meet with some success.

I fear that this is my ONLY option.

“Flood filling the transparent texels in a texture image with colors to match the opaque portion will fix texel interpolation but this may not be required depending on the source image.”

When is that benificial and when is it not?

If you look at the code above you will see all of the things I have tried. None of them worked with any good result. Please just tell me if I missed something I can try or maybe I have expended all other options except flood filling.

Flood filling helps when the color in the transparent region doesn’t match the color in the opaque region. As the texture alpha samples are interpolated to transparent the color is interpolated from the correct opaque color to whatever incorrect color is under the transparent alpha. The transition region has semi-transparent alpha and color that diverges from the desired color producing a halo. This happens when you switch from a nearest filter to a bilinear filter for example.

Someone on gamedev posted floodfilling/bleeding code for you a while back.

Just take your texture and for each transparent sample, find the nearest opaque sample and match its RGB color. You can do this recursively and just look for adjacent opaque pixels, the results will please you.