Using stencil to update stencil

[Lindley]

I'd like some details on precisely how stencil and zbuffer culling work.

In particular, let's say I've masked out part of an image in one pass, and I'd like to mask out more of it in the second.

What I'd like to do is, on the second update, only bother spending processor time running fragment programs on those places which were not previously masked out.

glStencilFunc appears to only control whether *color* is written in a given location----not whether anything at all is attempted there. So that isn't quite what I want. I figure this since you can set the stencil buffer up even with GL_NEVER set as the func. (Unless, of course, that's simply making use of the stencil fail case in glStencilOp...?)

Early-z may be useful, since that seems to prevent fragment programs from running entirely. However, I can't update the zbuffer this way, because doing so will disable early-z!

So I'm thinking I may need to do some sort of joint early-z/stencil approach, where each is updated by the other in turn.

What I need to know is where in the pipeline, precisely, each of these tests aborts rendering of a given fragment.

[dorbie]

The pipeline defines z and stencil after shading.

Hardware doing early z reorders this and MAY defer writes pending fragment kills or alpha test, or may not (disabling early z if there is alpha test for example) depending on the design.

Z may also be rejected on a tiled basis before fragment interpolation is even performed.

This is the reason early z is such a significant design feature. So sorry, but the architectural diagram will not inform you about the exact order to code for.

It is a function of the detailed architecture design that varies from architecture to architecture and even varies by state, and may be disabled permanently in a scene until the next clear if you use post z pixel kills. The early z and no early stencil is exactly the kind of platform variability you have to live with. It may work on another card which will be a potential performance win for the designers who could be bothered to implement it.

Not the answer you were hoping for :-).

Sorting by eye z is the biggest win but don't overdo it, just draw some big conspicuous occluders first and don't spin your wheels doing it.