I am trying to figure out a way to cut out a certain region of a background texture such that a certain custom pattern is not rendered on the screen for that background. For example:
Click image for larger version. 

Name:	C8bEZ.jpg 
Views:	94 
Size:	3.9 KB 
ID:	774
This square can be any pattern. I am using Frame Buffer Object and Stencil Buffer to achieve this kind of effect. Here is the code:
Code :
fbo.begin();
//Disables ColorMask and DepthMask so that all the rendering is done on the Stencil Buffer
Gdx.gl20.glColorMask(false, false, false, false);
Gdx.gl20.glDepthMask(false);
Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 1, 0xFFFFFFFF);
Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
 
stage.getSpriteBatch().begin();
rHeart.draw(stage.getSpriteBatch(), 1); //Draws the required pattern on the stencil buffer
 
//Enables the ColorMask and DepthMask to resume normal rendering
Gdx.gl20.glColorMask(true, true, true, true);
Gdx.gl20.glDepthMask(true);
 
Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 1, 0xFFFFFFFF);
Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);
 
background.draw(stage.getSpriteBatch(), 1); //Draws the background such that the background is not rendered on the required pattern, leaving that area black.
 
stage.getSpriteBatch().end();
Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST);
fbo.end();
However this is not working at all. How am I supposed to do this using Stencil Buffers? I am also facing some difficulty understanding glStencilFunc and glStencilOp. It would be very helpful if anyone can shed some light on these two.

I have also tried producing this effect using glColorMask but to no avail. The following code should produce a heart with a hollow square (But is not). Here is the code for that procedure:
Code :
Gdx.gl20.glClearColor(0, 0, 0, 0);
stage.draw();
FrameBuffer.clearAllFrameBuffers(Gdx.app);
fbo1.begin();
Gdx.gl20.glClearColor(0, 0, 0, 0);
batch.begin();
rubber.draw(batch, 0);
Gdx.gl20.glColorMask(false, false, false, true);
coverHeart.draw(batch, 1);
Gdx.gl20.glColorMask(true, true, true, false);
batch.end();		
fbo1.end();
 
toDrawHeart = new Image(new TextureRegion(fbo1.getColorBufferTexture()));
batch.begin();
toDrawHeart.draw(batch, 1);
batch.end();

Note: I am using the libgdx library.