Frame buffer color keying

Hello,

I need to replace parts of the current frame buffer that have a specific color with a texture. In other words, I have a background texture which should “shine through” all black (or green, or blue) parts of the scene. So basically I need some sort of destination color keying. I cannot use custom pixel shaders, since the hardware does not support it.

I was hoping this is directly supported by OpenGL, but the only way I found is actually retrieving the whole buffer using glCopyPixels, modifying it manually, and storing it back using glDrawPixels, which of course is very slow.

Background: I’m working on an augmented reality extension to an existing (but black-box) rendering engine. I get a camera image from an HMD which should be enhanced with cg 3D models. These models should be realistically occluded by real world objects (like walls), so they’re drawn into a model of the real world. Now the problem is replacing the walls by the camera image.

This is very easy if one has access to the engine source: Clear the frame buffer with the camera image, render the walls to the stencil buffer only, and then render the visible models on top of it using stencil testing. Sadly, I do not have access to the engine source, so this is not the way to go. The basic layout of the code is as follows:

<free to change this part>
engine.paint(); // paints world and objects
<free to change this part>
SwapBuffers();

If you have any other ideas on how to do this, or know of a more elegant way to solve the problem, please let me know.

Thanks,
Jan

This depend on what control you have on the engine itself (and what state changes it uses internally)

I assume from the above that you can set the color of the walls rendered?

If the engine does not use/set alpha blending/alpha testing, what you could do is this:
-Ensure you have destination alpha render surface
-Set the walls with a alpha value of 0 (ensure alpha testing and blending is off and all other surfaces have an alpha of 1)

<render>

-After the render only the walls have a alpha of 0 so enable blending with ONE_MINUS_DST_ALPHA:ZERO and render a full screen quad with you camera image.

If you can also try this old thread:
http://www.opengl.org/discussion_boards/cgi_directory/ultimatebb.cgi?ubb=get_topic;f=3;t=007658
(assuming you have two texture support and a stencil mask -see the last post by zeckensack) You may also have to turn off color buffer writes when setting up the stencil buffer.

Thanks a lot for the fast and great answer! This (“abusing” the alpha channel) works perfectly and at top speed. I love this board, no bull****, just qualified and precise answers. Thanks again!