Shadow mapping problem

[Robotnik]

Right now I'm trying to implement shadow mapping into my engine, I thought I had problems with back projection, but it turn out that's not the issue here.
Right now I'm having what seems to be problems with the texture matrix I use to project my end-result textures, or the matrixes I use for projection and modelview when I render into my depth texture. When I render solely character models, their shadows display fine, but when I add the world to the shadow map, nothing displays at all, like everything was shadowed. I can't find the cause of this problem, but it's bound to be a problem with the projection/modelview matrix. I've tried hard to figure it out, but I can't find the reason. It seems like the depth values in my depth texture are somehow off, but I don't get how that is possible.

Here's how I set up projection and modelview:

float flSize = tan((M_PI/360) * m_pCurrentDynLight->cone_size);
float flFrustum[] = { 2/(flSize*2), 0, 0, 0, 0, 2/(flSize*2), 0, 0, 0, 0, -1, -1, 0, 0, -2, 0 };
int bReversed = IsPitchReversed(m_pCurrentDynLight->angles[PITCH]);
vec3_t vTarget = m_pCurrentDynLight->origin + (m_pCurrentDynLight->forward * m_pCurrentDynLight->radius);

// Set projection
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glMultMatrixf(flFrustum);

// Set modelview
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
MyLookAt(m_pCurrentDynLight->origin[0], m_pCurrentDynLight->origin[1], m_pCurrentDynLight->origin[2], vTarget[0], vTarget[1], vTarget[2], 0, 0, bReversed ? -1 : 1);


This is how I set texture projection for the depth texture:

glTexGeni(GL_S, GL_TEXTURE_GEN_MODE, GL_EYE_LINEAR);
glTexGenfv(GL_S, GL_EYE_PLANE, planeS);
glTexGeni(GL_T, GL_TEXTURE_GEN_MODE, GL_EYE_LINEAR);
glTexGenfv(GL_T, GL_EYE_PLANE, planeT);
glTexGeni(GL_R, GL_TEXTURE_GEN_MODE, GL_EYE_LINEAR);
glTexGenfv(GL_R, GL_EYE_PLANE, planeR);
glTexGeni(GL_Q, GL_TEXTURE_GEN_MODE, GL_EYE_LINEAR);
glTexGenfv(GL_Q, GL_EYE_PLANE, planeQ);

glEnable(GL_TEXTURE_GEN_S);
glEnable(GL_TEXTURE_GEN_T);
glEnable(GL_TEXTURE_GEN_R);
glEnable(GL_TEXTURE_GEN_Q);

// load texture projection matrix
glMatrixMode(GL_TEXTURE);
glLoadIdentity();

glTranslatef(0.5, 0.5, 0.0);
glScalef(0.5, 0.5, 1.0);
glMultMatrixf(flFrustum);

MyLookAt(m_vCurDLightOrigin[0], m_vCurDLightOrigin[1], m_vCurDLightOrigin[2],
vTarget[0], vTarget[1], vTarget[2],
0, 0, bReversed ? -1 : 1);

glMatrixMode(GL_MODELVIEW);

I don't see the problem in this code, but maybe its my lack of understanding of how these matrixes work. I'd apprechiate any help.

Thanks.

[Robotnik]

Okay, so I realised the problem was that I was setting glDepthRange to (0.0, 0.8). Now when I get rid of this I get a semi-normal result.

But I need this to be set, I have all normal objects in the 0-0.8 range while I have my skybox objects in the 0.8-0.9 range, and the skybox itself in the 0.9-1.0 range. How can I account for the depth range when I render shadows?

[BionicBytes]

Why not leave depth range as 1.0 and just disable depth writing for all sky objects? Do they really need depth?

[Robotnik]

Thanks for the reply. I didn't realise I don't need to render sky objects inside the shadow pass, so I just set the depth range to 0-1 there. Now another problem came up, which I have no idea what's causing it. Parts of the shadow seem to be disappearing and I have no idea why. Here are some pictures at two distances. There's a character model in that room, but I disabled his rendering so the problem is more visible.

http://oi40.tinypic.com/29db5sy.jpg
http://oi39.tinypic.com/akyhk9.jpg

My texture resolution is 512x512 pixels, my friend keeps telling me it might be a problem but I doubt it is. Any idea what is going on?

[Robotnik]

Okay, I fixed that problem. Problem was, I was thinking I needed to flip on on the y axis in the matrixes when I didn't need to, and thus I got false results on some stuff.

Now I have correct results on shadows, but wherever now the spot light texture shouldn't be shadowed, there's some very strange texture artifacts. Here's an image:
http://oi42.tinypic.com/4jxg0n.jpg

Anybody have any idea what this is caused by?

[BionicBytes]

Without seeing the actual object causing the shadow it's hard to see what's going on. To me it looks like you are not projecting the verticies properly and it looks like a silhouette drawn into the scene where the character should be.
Which projection matrix did you use?

[Robotnik]

The shadowing object there is solely the character, wherever he shadows it's fine. The problem is where the scene is not shadowed, this weird pattern occurs. Seems like a problem with the depth values, but I have no idea how I could fix this.

[Ed Daenar]

Have you tried implementing the projection and for the UV coordinates yourself instead of using glTexGen? It might help you diagnose where the actual problem is.

I find shadow mapping much easier to do when I'm in control of everything.