Detecting if a gl_LightSource is disabled in compatibility profile

I am writing a GLSL program as part of a plugin running inside of Maya, a closed-source 3D application. Maya uses the fixed function pipeline to define it’s lights, so my program has to get it’s light information from the gl_LightSource array using the compatibility profile. My light evaluation is working fine except for one thing, I cannot figure out how to determine if a particular light in the array is enabled or disabled. When Maya enables new lights my program works as expected but when Maya disables a previously enabled light it’s parameters are not reset in the gl_LightSource array and gl_MaxLights obviously does not change, so my program continues to use that stale light information in it’s shading computation. The light’s position, colors etc, all continue to have stale non-zero values after it is disabled. Maya draws all other geometry using the fixed-function pipline (no GLSL) and those objects correctly ignore disabled lights, how can I mimic this behavior in GLSL?

You can’t mimic it entirely in GLSL. You’ll need to query the state of each light in GL, via glIsEnabled(GL_LIGHT0 + light_index), and then assign that to a uniform array that’s declared in your shader:

uniform int light_enabled[ gl_MaxLights ];

I always found it a little odd that there are derived quantities in the GLSL builtin lighting state, but yet no ‘light enabled’ uniform. Pretty much a moot point now, though.

Circa 2004, hardware didn’t used to be near as flexible as it is now (in program length, number of uniforms, execution speed, etc.).

It was much less practical then to consider “do everything” shaders that took in a superset of uniforms to handle every possible conceivable case and dynamically decide (runtime branch) based on what specific shading algorithm you needed to implement (num lights, light types, cone Y/N, color material mode, fog mode, fog range mode, lighting equation, separate specular Y/N, two-sided lighting Y/N, local viewer Y/N, etcetc…).

And even in this day-and-age, it can still be a big waste of GPU resources doing this unless you carefully constrain it.

So I think there was just an implicit understanding that these “shader permutation” problems were handled by controlling what code was provided to the shader assembler, not by dynamic run-time branching on the GPU in the shader code.

However…

In hindsight I wish GLSL had provided some simple in-language support to help us manage this “shader permutation” problem. That is I wish (and still wish for it to be provided in the future) is for GLSL to explicitly require in the specification that all conditionals (e.g. “if” statements) evaluated on constant expressions be evaluated at compile time (never “at run-time”), throwing away any resulting “dead code” that results. Some vendors already do this, and do it very, very well. But this is not required by the spec.

“Why” you say? Then you can have something like this:


const bool gl_FogEnable;

if ( gl_FogEnable )
{
   ...
}

and you know the GLSL compiler will evaluate this “if” at compile-time, determine it is false (if fog is disabled), and throw away the if and all the code in the “if” branch, including references to any uniforms required for computing fog for this compiled shader. You could precompile the permutations you knew your app was going to use (fog off, fog exp2, etc.) and cache them off for use at run-time.

The very same thing applies when mapping this to modern OpenGL, except that there of course would be no built-in “const” shader permutation selectors you could use for this purpose (i.e. selecting which shader permutation you were compiling for). You just define your own set of “const” selectors and set them by either 1) hard-coding their values in the shader source (e.g. via sprintf) or (better) 2) by the GLSL API providing a clean way to set them (e.g. via the glUniform* API, but whose values are “plugged in” at compile-time; maybe call them "const uniform"s or something):


const bool FOG_ENABLE = false;

if ( FOG_ENABLE )
{
   ...
}

malexander, thanks, I’ll do that.

Dark Photon, I see what you are saying. As for your “wish”, I think it is possible. There is a pre-processor and the source supplied to the compiler is actually an array of strings, so you can tack on defines at compile time. I’ve been using the following method in my code:

C++:


    <snip>

    const GLchar* sources[2];
    GLint sourceLengths[2];

    // set pre-processor defines:
    const GLchar* defineFlat = "#define FLAT
";
    const GLchar* definePrecise = "#define PRECISE
";
    if (flat) {
        sources[0] = defineFlat;
        sourceLengths[0] = strlen (defineFlat);
    } else if (precise) {
        sources[0] = definePrecise;
        sourceLengths[0] = strlen (definePrecise);
    } else {
        sourceLengths[0] = 0;
    }
    
    // set main shader source:
    sources[1] = (GLchar*) glsl_bssGlHair_frag;
    sourceLengths[1] = glsl_bssGlHair_frag_len;
    
    // compile:
    GLuint fragShader;
    glShaderSource (fragShader, 2, sources, sourceLengths);
    glCompileShader (fragShader);

    // link, etc...

    <snip>

GLSL:


// GLSL FRAGMENT SHADER
#version 410 compatibility

#if defined PRECISE
# define FLOAT double
# define VEC2 dvec2
# define VEC3 dvec3
# define VEC4 dvec4
# define MAT4X4 dmat4x4
#else
# define FLOAT float
# define VEC2 vec2
# define VEC3 vec3
# define VEC4 vec4
# define MAT4X4 mat4x4
#endif

uniform vec4 viewport;

flat in vec3 segmentPosition[2];
flat in vec4 segmentColor[2];
flat in vec3 segmentNormal[2];

mat4x4 projMatInv = inverse (gl_ProjectionMatrix);
vec2 viewPitch = 2. / viewport.zw;
float depthSum = gl_DepthRange.near + gl_DepthRange.far;
float depthDiv = 1. / (gl_DepthRange.far - gl_DepthRange.near);
const float infinity = 1. / 0.;

void main ()
{
    // fragment position:
    vec4 clipPosition;
    clipPosition.xy = (gl_FragCoord - viewport).xy * viewPitch - 1;
    clipPosition.z = (2. * gl_FragCoord.z - depthSum) * depthDiv;
    clipPosition.w = 1.;
    clipPosition /= gl_FragCoord.w;
    vec3 fragPosition = (projMatInv * clipPosition).xyz;

    // precision casts:
    VEC3 p0 = segmentPosition[0];
    VEC3 p1 = segmentPosition[1];
    VEC3 pD = p1 - p0;

    // intersection results:
    vec3 position;
    float u;

#if defined FLAT

    position = fragPosition;
    float distance0 = distance (fragPosition, p0);
    u = distance0 / (distance (fragPosition, p1) + distance0);

#else

    <snip>

Hopefully that is self explaitory?

[QUOTE=Dark Photon;1239825]… GLSL API providing a clean way to set them (e.g. via the glUniform* API, but whose values are “plugged in” at compile-time; maybe call them "const uniform"s or something)
[/QUOTE]
This would be really useful. I also generate code with const variables just to do that.

I’d also appreciate to have the opposite uniform qualifier. Sometimes (NVIDIA) compiler is too eager at optimizing code. For example have a shader with if() condition using a float uniform. Sometimes I observe shader recompilation when the float value reaches some well known number, like 0.0 or 1.0. The problem is that if you animate such uniform and the value just passes this special value then perf. drop happens. This behavior is very unpredictable and very dependent on driver version.

The good thing is that these features are not HW related.

I think the driver shouldn’t try to be “too smart” in these kinds of situations as you might run into issues like the one presented by mfort. Everything should be explicit and the driver shouldn’t do tricks (like generating specialized shaders on the fly or renaming buffers and stuff like that). If the API provides all the functionality needed to express every need explicitly (in this case we already have preprocessor directives, as atb123 explained it) then the application developer could be able to use the API in an efficient and consistent way. “Too smart” drivers may be more efficient in some cases but they are not consistent and that could hurt you.
However, we still need a more explicit API for the presented approach to work better in practice than it currently does (just think about buffer usage flags).

Sometimes I observe shader recompilation when the float value reaches some well known number, like 0.0 or 1.0.

I’m new to GLSL so I’m probably missing something basic but this statement confused me. I thought compilation happened just once when you call glCompileShader? Isn’t it up to you to decide when the shader gets recompiled?

Well, glCompileShader probably does some compilation. But you never know what is the result of the compiler. It could be some intermediate code that gets compiled to the final GPU instructions when the shader is used for the first time.

I see, thanks.

Managing GLSL compilation sure does take a lot of code, I feel like I’m writing a mini make system. It isn’t hard, just cumbersome. Are there any good open source C++ libraries that can help with this?

[QUOTE=atb123;1239862]Dark Photon, I see what you are saying. As for your “wish”, I think it is possible. There is a pre-processor… I’ve been using the following method in my code:

C++:


    const GLchar* defineFlat = "#define FLAT
";

GLSL:


#if defined FLAT
    position = fragPosition;
    float distance0 = distance (fragPosition, p0);
    u = distance0 / (distance (fragPosition, p1) + distance0);
#else

[/QUOTE]

That was the first approach I used years ago. After you get a few shader permutation settings in there, the code gets to be an unreadable difficult-to-maintain mess!

It’s much cleaner (more readable, easier to maintain) to use "if/else"s in the shading language instead of #if/#elses with the preprocessor to accomplish this. The code actually reads completely normally then, and you can freely mix these constants in expressions with uniforms and other non-constant variables in the shader, and the compiler will just “do the right thing” when it comes to discarding unreadable code (inactive shader permutations).


  if ( FLAT && LIGHTING_ENABLED && LIGHTING_EQUATION == FULL )
  {
    position = fragPosition;
    float distance0 = distance (fragPosition, p0);
    u = distance0 / (distance (fragPosition, p1) + distance0);
  }

Really!? I haven’t had to hack-around that behavior since GeForce 7 days. What GPU/driver?

Yes, that is happening to me now. And now that I’m trying to add compile-time light type/enable switching…yuck. I’m spoiled by C++ templates I guess.

This does seem cleaner, but either way, not so much fun to manage. This is begging for a nice C++ library wrapper.

100% agreed. However, I am not talking about using standard uniforms for this. uniforms have a defined purpose. They are constant within a batch but can change across multiple invocations of that same compiled/linked shader program. The GPU+driver should never recompile/reoptimize your shader code when you change a standard uniform value – at least in this day-and-age. That was done ~7 GPU generations ago due to HW limitations, but I haven’t seen/heard any clear indications that this has gone on since.

No, what I’m talking about is using constants for this. Constants by definition cannot change across all invocations of that specific compiled shader source. They are unchangeable. And thus they are ideal for setting “shader permutation” state which the compiler can automagically use to snip away code for shader permutations that aren’t active.

For instance:


const bool FLAT = false;

if ( FLAT )
{
  ...
} 

When any compiler that’s worth 5 cents builds an expression tree and plugs in the constants, it effectively sees:


if ( false )
{
  ...
}

so all the code in the braces will never be executed for this compiled shader and can be thrown in the bit bucket, along with the “if” expression evaluation and branch as well. It knows this at compile-time.

So we get what we want, and all without any #if/#else/#elif/#endif mess.

I’m not sure I follow, didn’t you just suggest using:


#define FLAT false

if ( FLAT ) {   ...
}

It seems like the functionality you are asking for already exists but with slightly different syntax?

Agreed. It’s alot more cumbersome than it needs to be, particularly when you have 20-30 “shader permutation” variables parameterizing your code.

Using const shader permutation settings and standard GLSL language if/else/switch/etc. statements cleans up the GLSL code a ton (vs. nested #if/#else/#elif/#endif etc.)

But you still have the mess on the C++ side with sprintfing that big block that gets manually slapped onto the top of your shader programs. For this, what I’d prefer to see is something like this:

GLSL:


  const bool FLAT = false;       // False is just the default; C++ GL API can override this

  if ( FLAT )
  {
    ...
  }

C++:


  glConstant1i( "FLAT", flat );  
  glCompileShader( ... );
  ...

The compiler just “plugs in” the provided value for FLAT into the shader and compiles/links with it (and of course throws away all the “if (FLAT) { … }” code if FLAT was set to false. The above seems like it might be a pretty simple change.

There are of course a lot of possible variations here. For instance:

[ol]
[li] We could declare these special “C++ API-settable” constants “const param” in the shader, and then only allow “const param” settings to be settable via the GL C++ API. [/li][li] We could have these const param settings be settable after compile but before link (like subroutines) and provide a uniform-like API to query for the active const param settings and set them (e.g. instead of glGetProgramiv( … GL_ACTIVE_UNIFORMS … ), glGetActiveUniforms(), and glUniform*(), use glGetProgramiv( … GL_ACTIVE_CONST_PARAMS … ), glGetActiveConstParams(), glConstant*().[/li][li]Or we could conceivably call these “const uniform” settings (instead of “const param”) and then reuse some of the GL C++ uniform API to set/get them (e.g. glUniform*). The tradeoff is potential confusion with the original “uniform” semantics (…but I think shader subroutines may have already muddied that water). [/li][/ol]

Managing conditional re-compiling also seems a bit tiresome, regardless of how the consts get passed in. I’m not writing quite enough GLSL code to justify writing a library for it, but I almost feel like I should. Does your suggestion eliminate the need for re-compilation?

Yes, I understand. But you couldn’t select e.g. between different uniform block definitions or interpolation methods using constant ifs. For me, preprocessor directives seem like a better alternative, though agree that drivers should be able to optimize out constant ifs too.

It was quite recently with Quadro 6000. Driver 295.73 fixed this problem.

For the large portion of cases (ordinary uniforms, interpolators, etc.) there’s no need for any conditional compilation (uniforms become inactive, unused interpolators go away, etc.). But yes, I agree if you’re trying to do something the language doesn’t support in the same shader source, you might need the preprocessor to force it all into the same file. Before you do though, it’s definitely worth considering whether you should just use a separate ubershader source file for that permutation – could be the cleaner approach.

2 more questions:

  • How is this idea different than branching on uniform bools? Couldn’t the compiler optimize uniform branches in the same way?

  • Previously I was under the impression that branches based on uniforms were relatively efficient, how much of a performance boost might I get using const branches set up by the preprocessor instead? For example, the branches required to handle light types and enable/disable states in the gl_LightSource array. Assuming gl_MaxLights is 8, there is quite a bit of branching required. I know the correct answer will be “test and see” but I would rather not spend the time if the expected impact is minimal.