Abusing the depth buffer

(this might be an advanced topic, but because I’m an intermediate programmer I’ll put it here for now.)

What I want to do is draw a shape, and write to gl_FragDepth so that it “erases” the existing depth values in that shape’s region, setting them to what they would be if I had called glClear(GL_DEPTH_BUFFER_BIT). HOWEVER: I only want to erase the depth values that are behind the shape that I’m drawing. I want the depth values in front of the shape to remain as they are.

In theory, this is simple:

  1. Ensure that the depth test is enabled.
  2. Draw the shape, testing fragments against the depth buffer
  3. If the fragments pass the depth test, proceed to write a uniform value to gl_FragDepth

The problem is that depth testing occurs after fragment processing, which means there’s no way to tell whether a fragment has passed the depth test from within the fragment shader. Even on hardware that allows early depth testing, as soon as I try to write gl_FragDepth, the early depth test is disabled. Even if I explicitly request an early depth test, I can’t write to gl_FragDepth!

Has anybody here had any luck trying something like this? Is it possible? What might be some workarounds?

Is it possible to read the value that’s currently inside the depth buffer from within the fragment shader? Then I could do the depth test myself to see whether or not it should be overwritten… (Framebuffer Objects, maybe?)

Thanks in advance!

Two ideas:
[ol]
[li]use the stencil test as follows:[/li][LIST=1]
[li]Make sure stencil buffer is all 0’s. Set stencil test to GL_EQUAL and reference is 0. Set stencil ops to:[/li][LIST=1]
[li]keep on stencil fail[/li][li]keep on stencil pass and depth fail[/li][li]increment on stencil pass and depth pass[/li][/ol]

[li]Draw your geometry usually[/li][li]Set stencil test to GL_EQUAL and reference is 1. Set depth test to GL_ALWAYS. Draw your geometry with gl_FragDepth funkiness[/li][li]dont’ forget to reset stencil buffer back to 0, i.e. undo the damage done at step 1.[/li][/LIST]

[li]OR: use image_load_store on the depth buffer; you will need to make the FBO stuff, i.e. depth buffer is a texture AND disable depth test. Your frag shader will read the depth value and if the value is greater, then overwrite. There are going to be ickies on overlap though.[/li][/LIST]

You should set stencil to 1 instead of increment in step 1.1.3, otherwise it would fail if the shape overlaps itself.

I’m liking the idea of #2, since I plan to set up VBOs anyway. Do I have to worry about image_load_store not being available in OpenGL 3.x, though?

The biggest problem with idea #1, for me, is that I’m already using the stencil buffer for other things, and really only have 1 (maybe 2) bits of it to spare.

Considering I don’t need to increment, and can just set a bit, I might be able to make it work, actually…

You should set stencil to 1 instead of increment in step 1.1.3, otherwise it would fail if the shape overlaps itself.

If the shape draws overitself, the stencil value is already one and the stencil test is equal 0, so the stencil test would fail leaving the stencil buffer as 1. However, step 1.3 should also have that make stencil ops all GL_KEEP so that stencil values are constant.

I’m liking the idea of #2, since I plan to set up VBOs anyway. Do I have to worry about image_load_store not being available in OpenGL 3.x, though?

Image load store is a GL4 feature, GL3/DX10 hardware do not have it; image load store has nothing to do with VBO’s.

The biggest problem with idea #1, for me, is that I’m already using the stencil buffer for other things, and really only have 1 (maybe 2) bits of it to spare.

you can tweak the #1 to use GL_INVERT instead of INCREMENT and use ref and mask to use the bit you have free.

Sounds like I’m going with idea #1. Thanks, kRogue!