Color-key transparency for a texture image

I’m trying to write a function that color-keys a given texture image. For those of you not familiar with color-keying, you specify a certain color to remove from the image, and all pixels with that color are turned transparent. Right now I have this:


void color_key(int tex, unsigned int key)
{
	int w, h, x, y;
	
	//Retrieve pixel data
	glBindTexture(GL_TEXTURE_2D, tex);
	glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, &w);
	glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT, &h);
	unsigned int buffer[w][h];
	glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_INT, buffer);
	
	//Iterate through pixels -- if a pixel is equal to the key, set its alpha value to 0.
	for (x = 0; x < w; x++)
	{
		for (y = 0; y < h; y++)
		{
			if (buffer[x][y] == key) 
				buffer[x][y] &= 0xFFFFFF00;
		}
	}
	
	//Save the modified texture
	glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, w, h, GL_RGBA, GL_UNSIGNED_INT, buffer);
}

but this causes the program to hang/crash without an error. I also tried using a 1D array to store the image, but it did the same thing.
Commenting out the glGetTexImage and glTexSubImage2D makes it stop crashing, but of course the function does nothing after that.

Anyone have an idea what I did wrong?

OpenGL is trying to write whsignof(unsigned int)4 bytes (widthheightelement_sizenum_elements) into buffer.
From the way you are adjusting the pixels, you probably want to use GL_UNSIGNED_BYTE instead of GL_UNSIGNED_INT in your glGetTexImage call (41 bytes per pixel instead of 44 bytes per pixel).

Thanks Dan, that fixed the crashing problem. I also added masking to ignore the alpha channel (so any pixel whose RGB values are the same as the key should be removed, regardless of its alpha component). It doesn’t seem to be working though – it doesn’t do anything at all. Here’s what I have now:


void color_key(int tex, unsigned int key)
{
	int w, h, x, y;
	key &= 0xFFFFFF00;
	
	//Retrieve pixel data
	glBindTexture(GL_TEXTURE_2D, tex);
	glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, &w);
	glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT, &h);
	unsigned int buffer[w][h];
	glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
	
	//Iterate through pixels -- if a pixel's RGB is equal to that of the key, set its alpha value to 0.
	for (x = 0; x < w; x++)
	{
		for (y = 0; y < h; y++)
		{
			if ((buffer[x][y] & 0xFFFFFF00) == key) 
				buffer[x][y] &= 0xFFFFFF00;
		}
	}
	
	//Save the modified texture
	glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, w, h, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
}

I was under the impression glTexSubImage2D permanently changes the texture in memory. Is this not the case?

Or could it maybe have something to do with mipmapping? I don’t really know much about how that works, I just assumed that changing the full-detail version of the image would change all versions of it.

The other mipmap levels won’t be automatically updated when you update level 0, unless you’re using the legacy GL_SGIS_generate_mipmap extension.

See OpenGL tip: Generate mipmaps and Common Mistakes - OpenGL Wiki for a discussion on the preferred glGenerateMipmap vs the alternative gluBuild2DMipmaps / GL_GENERATE_MIPMAP methods.

As you suspected, If you’re updating level 0, but only higher mipmap levels are being chosen when drawing you won’t see the results until the object is closer to you. One way you can help visualise which mipmap levels are visible is to upload a different color to each level, see http://www.arcsynthesis.org/gltut/Texturing/Tut15%20Needs%20More%20Pictures.html for an example of this in action.

Should be:


	unsigned int buffer[h][w];

Similarly, the accesses should be buffer[y] and the loops should have the for-y loop outside the for-x loop.

It happens to work in this case because using a 1-D array of (w*h) elements would also work.

Although if the data wasn’t RGBA, you would either need to use glPixelStore() to change the pack/unpack alignment (the default is 4-byte alignment), or modify the loop to allow for padding between rows.

Dan: That is unfortunate…I can’t just go through all the mipmap levels, because the target pixels may have been blended with other pixels during the mipmapping. But it should work if I enable GL_GENERATE_MIPMAP, correct? (I’m using a legacy version of OpenGL – whatever came with my Windows 8.)
I will give it a try.

GClements: Really? I’m surprised that OpenGL would use column-major order instead of row-major. Well, I can easily switch that around, but as you said it doesn’t actually effect anything since the size of the array doesn’t change. It’s just a little misleading about where those pixels actually are in the image.

OpenGL stores textures in row-major order, which is why the existing code is wrong.

Maybe you’re confused about C array syntax? C doesn’t have multi-dimensional arrays, but it does have arrays of arrays.

The declaration


int buffer[h][w];

declares “buffer” as an array of h elements, each if which is an array of w elements, each of which is an int.

So for any integer i, buffer[i] is an array of w ints (i.e. a “row”).

For any integers i and j, buffer[i][j] is equivalent to (buffer[i])[j], i.e. element j within buffer[i], i.e. the cell in row i, column j.

Maybe you’re confused about C array syntax? C doesn’t have multi-dimensional arrays, but it does have arrays of arrays.

I know C stores arrays in row-major order, but it considers the first index as the row and the second index as the column.

An array declared in C as
int A[2][3] = { {1, 2, 3}, {4, 5, 6} };
is laid out contiguously in linear memory as:
1 2 3 4 5 6

From the wikipedia article, ‘Row-Major Order’. So the most efficient way to iterate through a 2D array is:


for (x = 0; x < w; x++)
    for (y = 0; y < h; y++)
        array[x][y] ...

Because you will access the elements in the order that they are stored in memory.

I simply use x for the first parameter, making x the row and y the column, because that seems intuitive to me and is the way I was taught.

Does OpenGL store x as the column and y as the row instead? I can see how this could be problematic in situations where I need to know the actual positions of pixels. In this case it still works either way, since I just need to modify every pixel regardless of where it is.

In which case, you’re probably the first person I’ve ever encountered who uses that, or who would find it intuitive. By itself, I suppose that it’s not necessarily wrong, but the universal convention is that X represents horizontal displacement while Y represents vertical displacement. Certainly, all of the OpenGL documentation uses this convention (e.g. glViewport).

Also, if x is the row and y the column, then the loop tests require that w is the number of rows (height) and h the number of columns (width). But those names are conventionally abbreviations for width and height respectively. And aside from convention, that’s how they’re being used in the glGetTexLevelParameter() calls in your original code.

If you swapped w and h in those calls, to:


	glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, &h);
	glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT, &w);

then the code would be correct, at least in a formal sense. The variable names would confuse practically every other graphics programmer on the planet, but there wouldn’t be anything else wrong with it.

Side note - pulling the pixels to the CPU and doing the work then pushing it back is a really poor idea. Use a fragment shader instead.

Bruce

Hi Bruce, I was hoping it would be okay if it is only done when initially loading a texture – which occurs one time in main and then never again. I don’t know how to use shaders yet though this is something I am planning to learn eventually.

Sorry I haven’t responded in a while, I never could get my code to work. I realized that I’m accessing the components backwards, it’s actually stored as ABGR. But it’s still behaving very oddly – it allows me to change certain components of a pixel but changing alpha doesn’t seem to do anything even though I’m using an alpha test. Also, no pixels are ever passing the key comparison :confused:

Will post updates if I ever figure this out.