Accessing the stencil buffer from shader would be a great idea. I’m not sure how expensive it would be with current hardware silicon capabilities, but it would be great to have it in mind for next versions of GL.
in/out gl_StencilBuffer[x][y]
so that a shader can read and write to this buffer in both vertex and fragment shaders.
While reading from various locations might work (perhaps via a GLSL function), writing to a sample other than the one the fragment shader is processing would likely be very problematic due to race conditions. Since you can’t do that with any other outputs from a fragment shader, I would think the chances are slim that it would be supported for stencil output.
GL_ARB_shader_stencil_export at least allows you to write to the current sample in the stencil buffer.
But if only enable writing to stencil buffer when stencil write fixed functionality is disabled explicitly. i.e,
glEnable(GL_PROGRAMMABLE_STENCIL_WRITE)
Otherwise it has no effect in shader if fixed functionality enabled, which is the default.
Getting access to any pixel of the stencil buffer from a shader is not going to go well: a GPU typically processes multiple pixels in parallel at the same time.
However there is a simple way to emulate this behavior: GL_EXT_shader_image_load_store , rather than the stencil buffer, you make an integer texture and manipulate that. This requires GL4 hardware and it is supported by both NVIDIA and AMD cards. the catches of this approach is that it is more expensive than built in stencil hardware and you need to handle the synchronizations issues carefully (it does support atomic operations and barriers).
Additionally, getting just the value of the stencil buffer of the fragment being processed by the fragment shader is also shaky, simply because many GPU’s handle the depth and stencil buffer at the “same time” [think how glStencilOp makes stencil operations interact with depth operations]. Also, getting the previous value of the depth buffer is not possible either, simply because is most use cases how the depth buffer is used is covered fixed function pipeline functionality, there are cases where you want more, and again, then GL_EXT_shader_image_load_store is the way to go in those cases.
The extension GL_ARB_shader_stencil_export lets one generate the “reference” value used in the stencil test from within the fragment shader. The catch here is that it has the same effects of stopping early testing just as setting gl_FragDepth.
If you can read and write to a buffer, then what is the point of calling it a stencil buffer?
You could just attach a secondary color buffer and read it, do your stencil test in your shader, and write to that buffer.
You expect to call glDisable(GL_PROGRAMMABLE_STENCIL_WRITE) and your shader will no longer allow you to read or write to gl_StencilBuffer?
That seems rather inconsistent. There isn’t a glDisable(GL_PROGRAMMABLE_DEPTH_WRITE). It’s your job to change shaders.