How the stencil buffer is working

OpenGL OpenGL: Basic Coding
1

HI all,

I ‘ve difficulty with understanding the logic of stencil buffer, the code bellow is supposed to carve out solid sphere from octagon. But what I got as result is hole in the octagon w/o depth perception, even if I look(rotate) from back of octagon, sphere is still visible, where it’s supposed to be hided by octagon geometry.

I’ve also provided my inline comments my way interpretation of stencil buffer.

Any comments will be appreciated.

Void display(void)
{
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);  

glEnable(GL_LIGHTING);
glPushMatrix();
glCullFace(GL_BACK); // DISCARD THE BACK FACE 

glStencilFunc(GL_LESS,1,1);//FRAGMENTS COMING FROM SPHERE WILL PASS THE STENCIL TEST 
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); // PASSING FRAGMENTS WILL MODIFY THE BUFFER CONTENTS WITH 1'S 
glPushMatrix();
glTranslatef(0.5,0,0);		
glColor3f(0,0,1);
glutSolidSphere(0.6,16,16); // DRAWING METHOD SPHERE 
glPopMatrix();		
// SO FAR, I'LL GET SOMETHING LIKE A MASK(OF SPHERE) IN STENCIL BUFFER 

//glDepthFunc(GL_ALWAYS);
glColor3f(1,0,0);
glStencilFunc(GL_EQUAL,0,1);//LET ONLY THE 0's PASS THE STENCIL TEST
glutSolidOctahedron();	

glCullFace(GL_FRONT);
glColor3f(0,0,1);
glStencilFunc( GL_NOTEQUAL,1,1);//STENCIL TEST THE SPHERE TO GET THE OCTAHEDRON INNER-FACE FILLED 
glPushMatrix();
glTranslatef(0.5,0,0);		
glutSolidSphere(0.6,16,16); // DRAWING METHOD SPHERE 
glPopMatrix();				
glPopMatrix();		
glutSwapBuffers();  
}

void init(void)
{
……………………  
glClearStencil(0); // ISN'T THAT EQUAL WITH FILLING THE STENCIL BUFFER WITH 0'S ??
glEnable(GL_STENCIL_TEST);  // ENABLE STENCIL TEST 
glEnable(GL_CULL_FACE);    // ENABLE FACE CULLING 
glEnable(GL_DEPTH_TEST);
glStencilFunc(GL_ALWAYS, 0, 1 ); // SET THE TEST COMPARISION FUNC TO FILL WITH 0'S  
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE); // FILL UP THE STENCIL BUFFER WITH 0'S 
}
2 

glStencilFunc(GL_LESS,1,1);//FRAGMENTS COMING FROM SPHERE WILL PASS THE STENCIL TEST 
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); // PASSING FRAGMENTS WILL MODIFY THE BUFFER

…no it won’t modify the stencil buffer. That’s becuse you have told stencilOp to KEEP the buffer contents when the fragment passes the depth test.

void glStencilOp( GLenum sfail,GLenum dpfail,GLenum dppass);

For each object rendering to stencil buffer, check carefully what you have setup for the depth test and stencil test.

3

Changed it to, but still the same
glStencilFunc(GL_LESS, 1,1); // FRAGMENTS COMING SPHERE WILL PASS THE STENCIL TEST
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE); // PASSING FRAGMENTS WILL MODIFY THE BUFFER CONTENTS WITH 1’S

4

there’s something not right with you usage of the stencil functions…
The trouble I’m not even 50% sure I know what you have in mind.

This is what I think you are trying to do by reading your posted code:

  1. Set stencil to 1 for all pixels of the sphere
  2. Draw Octahedron where stencil buffer pixels are =0 - ie not where the sphere was drawn
  3. Draw another sphere on top of the Octahedron where stencil pixels are <> 1 ???

however, I’m not really sure that will even work. Can you draw a simple picture (ms-paint) and show us the areas covered by the objects and the values you expect to be set in the stencil buffer by each?

As I said before, you need to set the stencil function and operation carefully for each object becuase the stencil buffer will potentially get altered depending upon the depth test AND stencil test for that pixel.

So, for example:


glStencilFunc(GL_LESS,1,1);//FRAGMENTS COMING FROM SPHERE WILL PASS THE STENCIL TEST 
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE); // PASSING FRAGMENTS WILL MODIFY THE BUFFER CONTENTS WITH 1'S 
glPushMatrix();
glTranslatef(0.5,0,0);		
glColor3f(0,0,1);
glutSolidSphere(0.6,16,16); // DRAWING METHOD SPHERE 
glPopMatrix();

well…lets see. I don’t think you are actually setting the stencil at all - at least not in the way you are expecting.
Firstly you have not defined what the depth test function actually is - do we assume GL_LESS ?

You need to understand that

stencilFunc(GL_LESS ,1,1)

means "compare the reference value “1” to the value of the stencil buffer (initially cleared to 0) and if “1” is less than “0” do something --> the stencilOp tells GL what to do next.
The way you have stencilfunc as GL_LESS means “Becuase “1” is not less than “0” we fail the stencil test and replace the stencil buffer with “1” anyway” which is due to your first parameter of glStencilOp(GL_REPLACE,…)

Your last post suggested you have glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE) which now means stencil values can be replaced with the reference value if either 1. the stencil compare fails, 2. the depth test fails or 3. both pass.
This is wrong. You should be saying replace only the stencil buffer values when the depth test passes, keep otherwise.
And for that you need glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
Additionally, because this is the first object drawn and you want to set the stencil (force it to contain “1” for all pixels of the sphere) you need to use

glStencilFunc(GL_ALWAYS,1,1)

Therefore your first sphere code should be something like:


glStencilFunc(GL_ALWAYS,1,1);//ALWAYS PASS THE STENCIL TEST 
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); // PASSING FRAGMENTS WILL MODIFY THE BUFFER CONTENTS WITH 1'S 
glPushMatrix();
glTranslatef(0.5,0,0);		
glColor3f(0,0,1);
glutSolidSphere(0.6,16,16); // DRAWING METHOD SPHERE 
glPopMatrix();

I’m not sure what you intend, so the following is just an example.

For the second object the Ocahedron - let’s say you only want to draw it where the stencil = 0 - ie not overlapping the sphere
for that you need:


glStencilFunc(GL_EQUAL,0,1);  //TEST FOR THE BACKGROUND VALUE 0; DONT DRAW OVER IT
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); // PASSING FRAGMENTS WILL MODIFY THE BUFFER CONTENTS
glutSolidOctahedron();

I’m not sure what’s supposed to happen with the 3rd sphere - but once you get the idea that you can draw an object and tag the stencil buffer, you can then test the reference value against the stencil buffer and the draw operation proceeds or fails depending.
Notice the operation is not the otherway round - which often trips up people. I.e. It is not that the stencil buffer is compared to the reference.

5

First of all thank you in advance,


Actually this is what I want to achieve. (Just assume the red cube is octahedron). Carve out sphere from octahedron. So the octagon will have recessed like shape.

This is what I think you are trying to do by reading your posted code:

  1. Set stencil to 1 for all pixels of the sphere
  2. Draw Octahedron where stencil buffer pixels are =0 - ie not where the sphere was drawn

Correct, the 3rd object(2nd sphere) is redundant here, I thought I can use 1st sphere only once (weird !!).
I’m completely wrong with that logic, ref values is compared to stencil buffer value not the vice versa which is clearly depicted in red book and OpenGL reference (my bad again).

well…lets see. I don’t think you are actually setting the stencil at all - at least not in the way you are expecting.
Firstly you have not defined what the depth test function actually is - do we assume GL_LESS ?

yes the initialization and depth “glDepthFunc(GL_LESS)” function is like that.

lClearStencil(0); // ISN'T THAT EQUAL WITH FILLING THE STENCIL BUFFER WITH 0'S ??
  glEnable(GL_STENCIL_TEST);  // ENABLE STENCIL TEST 
  glEnable(GL_CULL_FACE);    // ENABLE FACE CULLING 
  glEnable(GL_DEPTH_TEST);
  glDepthFunc(GL_LESS);
  glStencilFunc(GL_ALWAYS, 0, 1 ); // SET THE TEST COMPARISION FUNC TO FILL WITH 0'S  
  glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

Althought your reviewed code post looks logical and consistent I still get that final image. I’m not even messing with mask value so I leave everywhere intact “1” for mask, I think that I’m making logical mistake trying to achieve recessed effect just only with 1 pass stencil.
I’m expecting the sphere stencil values to be filled with “1” and the other all stencil values to be filled with “0”.

6

Looking at the cut-out sphere and cube images, I now realise what you want to do and this won’t be easy.
The cube must be drawn with back-face culling switched off to start with - because we are looking at the ‘blue’ inners of it.
Also, becuase the back facing inners are blue - you need two sided colouring.
If you can get that bit working - ie move the camera around the scene so it passes into and through the cube - you can then check that you have a blue interior.
Only at this point the shoudl you introduce the stencil asmked sphere as I directed earlier.

7

is there any simple demo for that ?

8

http://www.opengl.org/resources/code/samples/advanced/advanced97/notes/node11.html

and more recent:

http://www.bluevoid.com/opengl/sig00/advanced00/notes/node23.html

9

I’m working with that hoary example CSG 8.4.3 Difference in official OpenGL tutorial page.

I basically understand the logic of stenciling to some extent, but couldn’t see the clear effect of extracted parts( i.e. from different vantages) bcoz my rotation relies on glRotate/glTranslate command I’m trying to find a more elegant way of viewing such as gluLookAt command.

10

Sorry to broach this issue again but but still couldn’t figured out how to get that working properly. With the following simplified code snippet the stencil buffer contents masking whole scene regardless of rotations, translations etc… made to the objects.

Changing the vantage point (e.g. with gluLookAt(0.3*(x-mousex),0, 4 , 0, 0, 0, 0, 1, 0)) I’m expecting to see the content of stencil buffer is modified appropriately regarding the octa and sphere positions/rotations on scene.

Intuitively I’m missing something obvious and I couldn’t spot it, any comments will be appreciated.

void display(void)
{
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);  	
glColorMask(0,0,0,0); 
glClearStencil(0);
glEnable(GL_DEPTH_TEST); 
glCullFace(GL_FRONT);
glPushMatrix();
  glTranslatef(1,0,0);
  glutSolidSphere(0.6,16,16); 			
glPopMatrix();

glEnable(GL_STENCIL_TEST);
glDepthMask(0);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
glCullFace(GL_BACK);
glutSolidOctahedron();

glStencilOp(GL_KEEP, GL_KEEP, GL_DECR);
glCullFace(GL_FRONT);
glutSolidOctahedron();

glStencilFunc(GL_NOTEQUAL,0,1);
glStencilMask(0);
glColorMask(1,1,1,1);
glDisable(GL_DEPTH_TEST);
glCullFace(GL_FRONT);
glColor3f(0,0,1);	

glPushMatrix();
  glTranslatef(1,0,0);
  glutSolidSphere(0.6,16,16); 			
glPopMatrix();

glutSwapBuffers();