GLSL - cube shadows - projecting

[donmof]

Before i ask, i would like to apologize for starting here with a question.

The thing is that i couldn't find answer to my problem anywhere, and decided to post it here finally, to speak to gurus.

Here is what i am trying to achieve:

Omni directional shadow, by using FBO and GL_TEXTURE_CUBE_MAP with GL_DEPTH_ATTACHMENT.

So far, all works fine - render to all 6 faces of the cube map within FBO looks promising, the only thing left is to project that within the shader.

Now, to visualize what exactly is happening:

- here is the screen shot from the level editor, you can see a single omni light there that is the shadow-caster.



- here is the scene rendered with light (without shadow)



- here is the gdebugger cube map dump





With all that, i am passing the cube map to the fragment shader using:

uniform samplerCubeShadow shadowMap;

I sample it in fragment with:

float SHADOW=shadowCube( shadowMap, normalize(lightToSurface) );
gl_FragColor=light(0)*SHADOW;

where

lightToSurface = ((gl_ModelViewMatrix * gl_Vertex).xyz-gl_LightSource[0].position.xyz) / gl_LightSource[0].spotExponent;

spotExponent - is the radius, i pass the Omni light radius with that to the shader.


This produces :





The question is:
How to project the shadow by using samplerCubeShadow in GLSL ?


Many thanks for any help with this, I've spent days trying to figure that out, no success.. What's interesting is that even google does not pop up any example .. is this samplerCubeShadow something that is rarely used, or not used at all?

[Alfonse Reinheart]

float SHADOW=shadowCube( shadowMap, normalize(lightToSurface) );

First of all, where does "shadowCube" come from? It's not part of the unextended standard GLSL (any version, even pre-1.30). So what extension provides it? Have you activated that extension?

Second, what values do you expect "shadowCube" to return? If it's a float on the [0, 1] range that is proportional to the number of samples that pass the depth comparison test, then you seem to have forgotten a key ingredient:

The depth comparison reference value.

I can't say how "shadowCube" works, but in 1.30 and above, "texture" when given a "samplerCubeShadow" sampler, takes a vec4, not a vec3. The fourth component is the reference value for the depth comparison. That is, the depth of this fragment, relative to the light source in question.

[donmof]

Alfonse Reinheart,
you are totally right, the 'shadowCube' was something i was told when i was speaking to few guys around.

And seriously I'll try hard to go back with the histories and find out who and how was trying to use it.

*update* found it,

http://www.opengl.org/registry/specs...pu_shader4.txt


OK, let's switch to 'texture' when using samplerCubeShadow.
How should that be used then, can you help me out _please_?

I have totally lost the plot now, what should go in vec4 for texture and what is returned back ?

Thank you v.much.

[Dark Photon]

shadowCube' was something i was told when i was speaking to few guys around.

Yes, I believe that's the name of the texture access function in GLSL 1.2 and earlier (more on that below). samplerCubeShadow is the name of the sampler type.

OK, let's switch to 'texture' when using samplerCubeShadow.
How should that be used then, can you help me out _please_?

How to project the shadow by using samplerCubeShadow in GLSL ?

I have totally lost the plot now, what should go in vec4 for texture and what is returned back ?

As EXT_gpu_shader4 indicates, coord.stp (i.e. coord.xyz) is the direction vector (from the point light to the fragment you are trying to evaluate shadows for). And coord.q is the R value (i.e. depth value to compare against).

Typically with hardware shadow lookups like this, the R value is the lights' window-space depth value (0..1) associated with the fragment you are computing shadows for. However, I'm a little unsure about with cube maps because there are 6 distinct light frusta/projections. I'll look around a bit. My guess is you use the direction vector (coord.stp) to figure out which face of the cubemap you're on and use its camera-eye-to-light transform to back-project into the right light space yielding coord.q (the depth value of the fragment in the light's window-space).

When you do a shadow lookup, the hardware does the depth compares and returns back to you a "visibility" term in 0..1, where 0 = no visibility (i.e. totally in shadow) and 1 = 100% visibility (i.e. not in shadow at all).

If you set NEAREST filtering on the depth texture you bind to the shadow sampler, then you'll always get only 0 or 1. However if you enable LINEAR filtering, then this enables PCF shadow lookups, which means the hardware may do multiple shadow lookups, compares on each, and blend the results, giving you 0, 1, or some number in between. With LUMINANCE DEPTH_TEXTURE_MODE, this value is stuffed in .r, .g, and .b of the return value, so you can grab it from any of these.

By the way, this presumes you've got your texture attributes set up as follows (typical for shadow map lookups):

glTexParameteri( target, GL_TEXTURE_COMPARE_MODE, GL_COMPARE_R_TO_TEXTURE );
glTexParameteri( target, GL_TEXTURE_COMPARE_FUNC, GL_LEQUAL );
glTexParameteri( target, GL_DEPTH_TEXTURE_MODE  , GL_LUMINANCE );
glTexParameteri( target, GL_TEXTURE_MIN_FILTER  , GL_LINEAR );
glTexParameteri( target, GL_TEXTURE_MAG_FILTER  , GL_LINEAR );

Also note that pre-GLSL 1.3, your texture access function should be shadowCube (or shadowCubeProj -- more likely with point light source shadows which are perspective projection). Post-GLSL 1.3 it'd be texture (or textureProj) -- in GLSL 1.3 they stopped having different names for each type of sampler and just overloaded the texture access function names.

However in both cases, AFAIK, the sampler type you should use is samplerCubeShadow (though it's unclear whether GLSL 1.2 supported this). The shadow on the end of the name requires: 1) that you bind a depth texture to it, 2) depth compare (GL_TEXTURE_COMPARE_MODE) must be enabled on the texture.

That said, I haven't personally coded depth cubemaps yet so there may be some errors here.

[arekkusu]

Cube + Proj is not valid. Projection by q doesn't work for cubemaps.

Cube translates an .str vector to a face + .st offset; for non-shadow lookups q would be the length of the vector which is meaningless. For shadow, q is actually the reference value.

SamplerCubeShadow is not supported in GLSL 1.20 unless you enable GL_EXT_gpu_shader4.

[donmof]

Dark Photon:

Thank you for a great introduction into this, of course my texture is set up as follows, so it meets the requirements.

glGenTextures(1, &tDepthCubeMap);
glBindTexture(GL_TEXTURE_CUBE_MAP, tDepthCubeMap);
glTexParameteri(GL_TEXTURE_CUBE_MAP_ARB, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_CUBE_MAP_ARB, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_CUBE_MAP_ARB, GL_TEXTURE_WRAP_R, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri( GL_TEXTURE_CUBE_MAP, GL_DEPTH_TEXTURE_MODE , GL_LUMINANCE );
glTexParameterf( GL_TEXTURE_CUBE_MAP, GL_TEXTURE_COMPARE_MODE, GL_COMPARE_R_TO_TEXTURE );
glTexParameterf( GL_TEXTURE_CUBE_MAP, GL_TEXTURE_COMPARE_FUNC, GL_LEQUAL);

for (int face = 0; face < 6; face++)
glTexImage2D(GL_TEXTURE_CUBE_MAP_POSITIVE_X + face, 0, GL_DEPTH_COMPONENT32F , 512, 512, 0, GL_DEPTH_COMPONENT, GL_FLOAT, NULL);

My guess is you use the direction vector (coord.stp) to figure out which face of the cubemap you're on and use its camera-eye-to-light transform to back-project into the right light space yielding coord.q (the depth value of the fragment in the light's window-space).

That would mean, something like :

uniform samplerCubeShadow shadowMap;
...
float visibility=texture( shadowMap, lightToSurface );

where

...
lightToSurface=vec4(gl_LightSource[0].position-(gl_ModelViewMatrix * gl_Vertex));

That lightToSurface when passed to texture(...) should lookup the correct face of the cube, what is exactly that vec4's 4th component (.q) - how do i calculate it ?



Thanks to all of you guys, hopefully we can get that sorted out..


*update*

I made a small test here,
this is the actual light setup:




The depth map




And the shader's are as follows:

[vertex shader]

//made it vec3 - frag recreates it as vec4
varying vec3 lightToSurface;
vec3 vertexPos = vec3(gl_ModelViewMatrix * gl_Vertex);
lightToSurface = (vertexPos-gl_LightSource[0].position) / gl_LightSource[0].spotExponent;

[fragment shader]

vec4 lts=vec4(lightToSurface.x,lightToSurface.y,lightTo Surface.z,1);
float cDepth=texture( shadowMap, normalize(lts));
gl_FragColor=light(0) * cDepth;

'1' in the vector.w results:

vec4 lts=vec4(lightToSurface.x,lightToSurface.y,lightTo Surface.z,3);
float cDepth=texture( shadowMap, normalize(lts));
gl_FragColor=light(0) * cDepth;

'3' results




Of course what is strange too is that the 'shadow' or whatever it is there 'rotates' to camera view.

[Dark Photon]

Cube + Proj is not valid. Projection by q doesn't work for cubemaps.

Cube translates an .str vector to a face + .st offset; for non-shadow lookups q would be the length of the vector which is meaningless. For shadow, q is actually the reference value.

Good point. I wasn't thinking. With shadowCubeProj/textureProj on a samplerCubeShadow, we'd be out of vec4 texcoord components to play with, so there's no room for .xyzw and the reference value (5 values).

Also, googling around looks like most folks just use samplerCube and textureCube/texture (i.e. straight texture lookup; no hardware depth comparison) to sample cube depth maps for point light shadows for some reason. They don't get the free PCF doing it that way, so I'm not sure why, yet.

[donmof]

Almost there,




The problem i have now is related to view matrix,
when my camera is moved, shadows are moving along with camera view vector.

Anyways,

here's how shaders look now:


[vertex]

varying vec3 LightDirection;
varying float distance;

//Calculate light direction for fragment to lookup correct face
//*works*
vec3 ecPos = gl_ModelViewMatrix * gl_Vertex;
vec3 aux = vec3(gl_LightSource[0].position-ecPos);
LightDirection = aux;
float far=gl_LightSource[0].spotExponent;
float near=1;


//Calculate far/near planes
//*works*
distance = max(max(abs(LightDirection.x),abs(LightDirection.y )),abs(LightDirection.z));
distance = ((far+near)/(far-near)) + (1.0/distance)*((-2.0*far*near)/(far-near));
distance = (distance+1.0)/2.0;

gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;

[/vertex]

[fragment]

varying vec3 LightDirection;
varying float distance;

void main()
{
float cDepth=texture( shadowMap, ( vec4(LightDirection.x,LightDirection.y,LightDirect ion.z,distance) ) );
gl_FragColor= light(0) * cDepth;
}

[/fragment]


<object width="425" height="350"> <param name="movie" value="http://www.youtube.com/v/Y-Ii3SjhBeA"></param> <param name="wmode" value="transparent"></param> <embed src="http://www.youtube.com/v/Y-Ii3SjhBeA" type="application/x-shockwave-flash" wmode="transparent" width="425" height="350"> </embed></object>

I've seen guys doing that on a simple samplerCube, they pack the depth values when rendering into a color cube map instead of using depth maps from what i've seen. But that sounds like a 'backdoor' - why? If we have a samplerCubeShadow here..?

If i could fix the matrix.. i think that would be all..

[Dark Photon]

My guess is you use the direction vector (coord.stp) to figure out which face of the cubemap you're on and use its camera-eye-to-light transform to back-project into the right light space yielding coord.q (the depth value of the fragment in the light's window-space).

That would mean, something like :

uniform samplerCubeShadow shadowMap;
...
float visibility=texture( shadowMap, lightToSurface );

where

...
lightToSurface=vec4(gl_LightSource[0].position-(gl_ModelViewMatrix * gl_Vertex));

That lightToSurface when passed to texture(...) should lookup the correct face of the cube, what is exactly that vec4's 4th component (.q) - how do i calculate it ?

Reading some more on point light source shadows with cube depth maps, it's become apparent to me there are some subtleties I'm not intimately familiar with. I'd suggest you read these links:

* Omnidirectional Shadow Mapping (GPU Gems)
* Shadow Mapping Algorithms
* Deferred Shading in S.T.A.L.K.E.R. (GPU Gems 2)

the first one entirely, and the last two where they specifically talk about "cube" maps. Apparently there are several options here.

Seems radial distance is often stored in the cube map rather than cube face eye-space Z distance.

However, the way that occurred to me first is the most like directional light source shadow mapping (2D texture vs. cube texture). That is:

1. having 6 separate camera-eye-space-to-light-clip-space transform matrices (see below for a diagram),
2. you choose the right one based the fragment-to-eye-vector transformed back into the light's frame of reference, and
3. you transform your fragment position by that selected transform in turn to give you the light-space position in the frame of reference of that face and thus the depth value (i.e. reference value, i.e. coord.q) to pass to the shadowCube/texture call.

Diagram from here: http://www.paulsprojects.net/tutorials/smt/smt.html


This is I think what the STALKER link calls treating an omni light as 6 spotlights.

By the way, in your current code, that you're seeing the shadows rotate with your camera makes sense because you're using a vector in the camera's EYE-SPACE to lookup into the shadow map. You need to be using a vector that is relative to the frame of reference of the "light", not the "camera". That'll get you the right direction vector for the lookup. Then the only trick is getting the right depth value for the comparison. And that's apparently going to depend on which alg you use for generating your depth cubemaps. See the links above.

[donmof]

this is getting v.heavy...

Thanks photon, I'll try to implement what you've told,
will come back with results.. if i manage to finish this.