PDA

View Full Version : What is the best way to achieve this effect (having trouble with stencil buffer)



GavinNL
09-12-2015, 12:33 PM
Hi Everyone,

I'm not sure if this question would be considered beginner or advance. I did some googling around but couldn't find a good explanation of how to do this.

Essentially have an a tree of shapes. where shapes can have 1 parent, and multiple children. Eg:

Rectangle
---->Square
---------->Triangle
---------------->Octogon
---------------->Polygon
---------->Circle
---->Circle
---------->Triangle
---------->Square

I would like to traverse down the tree and draw each shape, but I need the parent shape to clip any child shapes. I've been experimenting with the stencil buffer, but I'm having some trouble with it. Is the stencil buffer even a good choice for what I need to do? I think I may be able to achieve this if I was able to do this with the stencil buffer:

"Draw shape where stencil buffer==x, if passes, replace buffer value with y"

But I don't think the stencil buffer has that option, all I can do is increase the buffer value by 1 using GL_INCR.

If anyone can provide some kind of pseudocode that can achieve that I need to do, that would be really appreciated.
Thanks

Alfonse Reinheart
09-12-2015, 02:10 PM
"Draw shape where stencil buffer==x, if passes, replace buffer value with y"

No, you can't do that. Not with stencil testing. The stencil operation that writes the stencil value only deals with two values: the current value from the stencil buffer, or the reference provided via glStencilFunc. You can't provide a third value to this process (unless `y` is always zero).

That being said, the overall goal is possible with the stencil test (https://www.opengl.org/wiki/Stencil_Test). It just requires a bit of work.

First, clear the depth buffer to 0. Also, initialize an integer to 0. Let's refer to this integer as `ref`. Also, enable the stencil test.

When you go to render any root object, do the following (passing all root objects a `ref` value of 0):

1) Set up your stencil test so that the stencil test passes if the stencil value is exactly <= to `ref`. And if it does pass, you want to increment the stencil value. Use this code:



glStencilFunc(GL_LEQUAL, ref, 0xFFFFFFFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);


2) Render the object.

3) Call every child object's rendering function, but the ref value you pass to them should be `ref` + 1. Each object at the same level of the hierarchy should get the same ref value (I know what you're thinking, and it's covered). This will obviously be recursive.

4) Here's the part that makes everything work. You now need to revert your stencil buffer back to where it was before this function was call. To do that, you want to re-render this object, without writing any colors (or depth values). But you do want to update the stencil value, changing it back to this function's version of `ref`.

However, you don't want the stencil test to always pass; you only want the stencil to be reset where the stencil value was changed by your object's rendering or any child objects' rendering. So you do this:



glWriteMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); //Turn off color writing.
glDepthMask(GL_FALSE); //Turn off depth writing.
glStencilFunc(GL_GEQUAL, ref, 0xFFFFFFFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);


5) Render this object again.

6) Turn color/depth masking back off:



glWriteMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); //Turn on color writing.
glDepthMask(GL_TRUE); //Turn on depth writing.


This method will work so long as the depth of your hierarchy of objects does not exceed the maximum value in your depth buffer (for 8 bits, 255).

Then again, you could do something similar with the depth buffer, so long as you can control the depth at which each shape gets rendered without modifying its mesh data.

GavinNL
09-12-2015, 03:31 PM
Thank you for your response, Alfonse!

I tried to implement what you suggested. I actually tried something like that before but ran into this same problem. Check out the screenshot below.

2086

The hierarchy of the shapes are as follows:

Green
----->Blue
----->Grey
--------->White

The white rectangle doesn't get clipped properly and ends up showing up in the blue rectangle as well. It seems to be dependant on the drawing order, if I reverse the hierarchy to

Green
----->Grey
--------->White
----->Blue

The white bar gets clipped properly.

Alfonse Reinheart
09-12-2015, 07:07 PM
The white rectangle doesn't get clipped properly and ends up showing up in the blue rectangle as well.

Based on the fact that the grey rectangle properly clips the white (at least, against the lower green rectangle), I'm guessing you fixed the bug in what I wrote (ie: use GL_EQUAL instead of GL_LEQUAL for the first pass) ;)

My guess is that you may have introduced a bug when you implemented it. For example, did you increment the `ref` variable itself when you drew the children, or just the value you passed to the child objects? Because if its the former, you still need the original `ref` value for the redrawing pass, when you set the stencil value back to neutral.

We'd need to see your code to be sure, one way or the other.


The white bar gets clipped properly.

No, it doesn't get clipped properly. You're simply overdrawing the white rectangle with the blue one.

GavinNL
09-12-2015, 11:44 PM
I think I managed to fix it. I replaced the GL_GEQUAL in the second pass with GL_LEQUAL and it seemed to work. Its getting late so I'll have to test it more thoroughly tomorrow. Thank you for your help!




Based on the fact that the grey rectangle properly clips the white (at least, against the lower green rectangle), I'm guessing you fixed the bug in what I wrote (ie: use GL_EQUAL instead of GL_LEQUAL for the first pass) ;)

My guess is that you may have introduced a bug when you implemented it. For example, did you increment the `ref` variable itself when you drew the children, or just the value you passed to the child objects? Because if its the former, you still need the original `ref` value for the redrawing pass, when you set the stencil value back to neutral.

We'd need to see your code to be sure, one way or the other.



No, it doesn't get clipped properly. You're simply overdrawing the white rectangle with the blue one.