PDA

View Full Version : OpenGL Shadows and the Stencil Buffer



aviel
05-10-2011, 06:12 PM
So, I've been following nehe's guide to creating shadows, and for whatever reason the shadows just aren't rendering. Not having an excellent understanding of things like the stencil-buffer means I'm sort of learning things as I go along. I hate outsourcing my debugging, but I'm kind of stumped here. does anybody know what's wrong?


glEnable(GL_CULL_FACE);
glCullFace(GL_FRONT);
glPushAttrib( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_ENABLE_BIT | GL_POLYGON_BIT | GL_STENCIL_BUFFER_BIT );
glDisable( GL_LIGHTING );
glDepthMask( GL_FALSE );
glDepthFunc( GL_LEQUAL );
glEnable( GL_STENCIL_TEST );
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
glStencilFunc( GL_ALWAYS, 1, 0xFFFFFFFFL );

glFrontFace( GL_CCW );
glStencilOp( GL_KEEP, GL_KEEP, GL_INCR );

//shadow volume quad drawing code here
//I've checked to make sure that the code to draw the volume is running
//so I thought I'd spare you that

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

glColor4f(0.0f, 0.0f, 0.0f, 0.4f);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glStencilFunc(GL_NOTEQUAL, 0, 0xffffffff);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

glPushMatrix();
glLoadIdentity();
glMatrixMode(GL_PROJECTION);
glOrtho(0, 1, 1, 0, 0, 1);
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();

glBegin(GL_QUADS);
glVertex2i(0, 0);
glVertex2i(0, 1);
glVertex2i(1, 1);
glVertex2i(1, 0);
glEnd();

glEnable(GL_DEPTH_TEST);
glPopMatrix();
glPopAttrib();

glEnable(GL_LIGHTING);
glDisable(GL_STENCIL_TEST);
glShadeModel(GL_SMOOTH);
glDisable(GL_CULL_FACE);


And I run that every frame. I have a feeling it's a problem in how I arranged the various gl*() commands. Only the large, screen-covering shadow is drawing with no parts being cut out.

vindy
05-10-2011, 06:50 PM
Hi there. I wanted to find out how to do shadows too and discovered how to do it with what's called a shadow matrix. I can give you the relevant code and at least partly explain how it works. I will admit though that it uses math from linear algebra which kind of above my head. I never took that course in college.

aviel
05-10-2011, 06:52 PM
I'm going to take that course eventually, but yes I'd love to see it. I don't know any linear algebra but perhaps I can compare that to my code and see what's going on, or use it instead or something.

Hoshang Kolivand
05-10-2011, 07:02 PM
I can help you but its first time I am using this page. give me you problem exactly I will help you
Hoshang Kolivand
shahinkey@yahoo.com

vindy
05-10-2011, 07:48 PM
Ok, put these two functions above your drawScene function:


void calcNormal(float v[3][3], float out[3])
{
float v1[3], v2[3], length;
static const int x = 0, y = 1, z = 2;

v1[x] = v[0][x] - v[1][x];
v1[y] = v[0][y] - v[1][y];
v1[z] = v[0][z] - v[1][z];

v2[x] = v[1][x] - v[2][x];
v2[y] = v[1][y] - v[2][y];
v2[z] = v[1][z] - v[2][z];

out[x] = (v1[y] * v2[z]) - (v1[z] * v2[y]);
out[y] = (v1[z] * v2[x]) - (v1[x] * v2[z]);
out[z] = (v1[x] * v2[y]) - (v1[y] * v2[x]);

length = (float)sqrt( (out[x] * out[x]) + (out[y] * out[y]) + (out[z] * out[z]) );

out[x] = out[x] / length;
out[y] = out[y] / length;
out[z] = out[z] / length;
}
void MakeShadowMatrix(GLfloat points[3][3], GLfloat lightPos[4], GLfloat destMat[4][4])
{
GLfloat planeCoeff[4];
GLfloat dot;

calcNormal(points, planeCoeff);

planeCoeff[3] = - ( (planeCoeff[0] * points[2][0]) + (planeCoeff[1] * points[2][1]) + (planeCoeff[2] * points[2][2]) );

dot = (planeCoeff[0] * lightPos[0]) + (planeCoeff[1] * lightPos[1]) + (planeCoeff[2] * lightPos[2]) + (planeCoeff[3] * lightPos[3]);

destMat[0][0] = dot - (lightPos[0] * planeCoeff[0]);
destMat[1][0] = 0.0f - (lightPos[0] * planeCoeff[1]);
destMat[2][0] = 0.0f - (lightPos[0] * planeCoeff[2]);
destMat[3][0] = 0.0f - (lightPos[0] * planeCoeff[3]);

destMat[0][1] = 0.0f - (lightPos[1] * planeCoeff[0]);
destMat[1][1] = dot - (lightPos[1] * planeCoeff[1]);
destMat[2][1] = 0.0f - (lightPos[1] * planeCoeff[2]);
destMat[3][1] = 0.0f - (lightPos[1] * planeCoeff[3]);

destMat[0][2] = 0.0f - (lightPos[2] * planeCoeff[0]);
destMat[1][2] = 0.0f - (lightPos[2] * planeCoeff[1]);
destMat[2][2] = dot - (lightPos[2] * planeCoeff[2]);
destMat[3][2] = 0.0f - (lightPos[2] * planeCoeff[3]);

destMat[0][3] = 0.0f - (lightPos[3] * planeCoeff[0]);
destMat[1][3] = 0.0f - (lightPos[3] * planeCoeff[1]);
destMat[2][3] = 0.0f - (lightPos[3] * planeCoeff[2]);
destMat[3][3] = dot - (lightPos[3] * planeCoeff[3]);
}

Now declare these at begining of drawScene function:

GLfloat shadowMat[4][4];
GLfloat points[3][3];

Set points to three points that lie on the plane where the shadow will be cast

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); //don't forget this


And then when you're ready to draw a shadow put this code before it with a light source of your choosing:

MakeShadowMatrix(points, lightPos0, shadowMat);

glPushMatrix();

glMultMatrixf((GLfloat *)shadowMat);

glColor3f(0.0f, 0.0f, 0.0f);

//draw the thing casting the shadow again

glColor3f(1.0f, 1.0f, 1.0f);

glPopMatrix();

aviel
05-10-2011, 07:59 PM
Set points to three points that lie on the plane where the shadow will be cast



Does the code rely on the fact that the shadow will be drawn on a single plane? Because it won't be.

vindy
05-10-2011, 08:43 PM
Umm, good question. I'm really not sure. Sorry. I thought I could be of help.

aviel
05-10-2011, 08:46 PM
Lol. where'd you get the code then?

vindy
05-10-2011, 09:24 PM
OpenGL Super Bible Second Edition. Found the book at the main library. Its been incredibly helpful so far. I'm new to OpenGL. :)

aviel
05-10-2011, 10:10 PM
Holy balls I need that book. Is there any code for projecting shadows onto something other than a plane?

vindy
05-10-2011, 10:44 PM
I'd have to dive into it again. However its 2am here right now and I'm beat. Gonna hit the sack. I'll follow up with you tomorrow.

aviel
05-11-2011, 12:35 AM
Alright, see you tomorrow, and thanks for the help. Hopefully this gets resolved because I've been debugging for a week and want to move on to particles and reflections.

I'm eventually going to be making a game based on the Israeli war of independence. maybe.

BionicBytes
05-11-2011, 02:36 AM
Holy balls I need that book. Is there any code for projecting shadows onto something other than a plane?

Yes it's called shadow mapping. There are loads of examples on the net and you'll find this technique better than planar stencil shadowing.
I also suggest you you shaders as this greatly simplifies the process of supplying the light projection * inverse scene camera matrix as opposed to using glTexgen commands.

aviel
05-11-2011, 12:21 PM
thanks, I'll take a look at that (although I wasn't using glTexgen commands to begin with for shadows)

BionicBytes
05-11-2011, 12:57 PM
You will do with shadow mapping because you need to compare the light source depth buffer (aka shadowmap) with the current cameras depth buffer. To do that you need to supply texture coordinates to index the shadowmap and to do that you need to send in the matrix consisting of lights projection matrix * lights camera * inv scene camera.

aviel
05-11-2011, 01:44 PM
Why should I do shadow mapping when a) I've already hacked out most of the shadow volume rendering and b) it's a bit less accurate?

Dark Photon
05-11-2011, 04:57 PM
So, I've been following nehe's guide to creating shadows, and for whatever reason the shadows just aren't rendering. Not having an excellent understanding of things like the stencil-buffer means I'm sort of learning things as I go along....
Well, on NeHe's site, it says:

Shadows:
This is an advanced tutorial. Before you decide to try out shadows, make sure you completely understand the base code, and make sure you are familiar with the stencil buffer.
So I'd suggest you drop back and try some simple stencil stuff with a 2D quad to get familiar with it. That's what I did when I was learning about it.

Before you beat your head against it too much more though, what are you shadow requirements? How many shadow casting objects? How many faces per object? Are they convex? Are they guaranteed to have no cracks? Can a shadow volume potentially intersect the near plane inside the frustum? How accurate do your shadows have to be? Do you need soft shadows? Are your lights and/or objects dynamic? Do you have area lights, or just point and directional lights? How big is your "world"? Do you need translucent or "fuzzy" objects (like volumetric smoke) to cast a shadow? What kind of CPU/GPU are bringing to the table? What res are you doing this at? Do you need sub-pixel accurate shadows? What frame rate do you have to maintain? etc.

You should be aware that that NeHe article is kinda old as shadow volumes tech goes, only describes a small subset of the basics of shadow volumes, and (as it even says) builds and renders shadow volumes very inefficiently. It describes Z-Pass shadow volumes (without calling them that), but doesn't detail the failure cases or what to do about them, and the perf tradeoffs of those options. And it doesn't talk about building shadow volumes more efficiently (the ancient Gamasutra article it points to for that doesn't do a whole lot better by modern tech standards, because it's 12 years old!)


Why should I do shadow mapping when a) I've already hacked out most of the shadow volume rendering and b) it's a bit less accurate?
It's all a matter of what your requirements are. Depending on your perf and quality requirements (and your graphics tech know-how at this point in your learning), shadow volumes might be just perfect for you and your needs, ...or they might be so completely wrong that it's not even funny.

Give us some more info about your requirements and we can help point you toward the path of most likely success.

aviel
05-11-2011, 09:19 PM
Sure. The end goal is to make a video game, although I recognize that's quite a long way off. However, the engine should be made with that in mind.

So, there will likely be a few hundred faces per object, convex and concave, often with cracks, and the shadows could easily reach the near plane. My shadows should preferably accurate to the pixel, and the objects will be dynamic though the light probably won't be. I will have a few light sources at any given time, mostly area lights but directional lights aren't out of the question. The world will be reasonably extensive: it certainly won't fit on just one or two screens. Translucent objects will probably cast a shadow (haven't decided), but I don't think I'll bother with particles and volumetric stuffs. For the really low end computers I'll just enable "circle under the model" shadows, but I expect anybody with something at the level of a 9800+ to be able to render shadows, although I'd prefer it to be less than that. As a game, this should be able to run at up to 1920 x 1080. They don't need to be sub-pixel accurate, but for the aforementioned hardware I'd like to maintain 30+ FPS.

If shadow volume rendering is an idiotic way to go then please tell me. I just want whatever works.

aviel
05-11-2011, 10:00 PM
Okay. So I forgot to tell glut to enable the stencil buffer. Luckily, your time isn't wasted because I'm still left with problems.

No shadows render. If I allow writing to the color buffer, then I can see that the shadow volume is rendering and that it's doing so correctly. However, it's not writing to the stencil buffer properly as, when I disable rendering to the color buffer, no shadows appear.

aviel
05-11-2011, 10:09 PM
And I got it. Well, not quite there are still some odd artifacts, but basically I got it.

aviel
05-12-2011, 11:22 PM
Looks like I have to bump this. Those "odd artifacts" don't appear to be going away.

I don't really know what's causing them but I do know that I want them gone. Is there a way I can disable self-shadowing using shadow volumes? For what it's worth, it appears that they appear mainly in concave areas.

BionicBytes
05-13-2011, 06:25 AM
For what it's worth, it appears that they appear mainly in concave areas.
Is that one of the weaknesses of this shadow volume technique?

aviel
05-13-2011, 02:09 PM
For what it's worth, it appears that they appear mainly in concave areas.
Is that one of the weaknesses of this shadow volume technique?

I thought it might be, but I really can't find it. And thinking of how shadow volumes work (extrusions of the silhouette), I don't see why these artifacts should be there.

Dark Photon
05-13-2011, 06:05 PM
Those "odd artifacts" don't appear to be going away. I don't really know what's causing them but I do know that I want them gone. Is there a way I can disable self-shadowing using shadow volumes? For what it's worth, it appears that they appear mainly in concave areas.
We're visual people. :p Let's see some pictures of the artifacts.

Like I mentioned before, there are limitations and constraints that shadow volumes imposes, and there are failure cases in that particular shadow volume technique that you need to get a solid handle on. If your artifact isn't a coding bug you've introduced, my guess is that one of those constraints is being violated or failure case is being exercised.

First, you mentioned your objects often have cracks. That's a big no-no. With shadow volumes, you have to be able to ensure that every pixel (or every sample, when rendering with MSAA) will be updated by every front facing and back-facing shadow volume face that should affect it -- no exceptions! To do that, you need to ensure that your extruded shadow volume silhouettes are perfect. And to ensure that you typically need a welded mesh. That is, a mesh where for each location there is exactly one vertex in the mesh representing it, and there are no cracks/T-junctions (all vertices on edges split that edge)! The silhouette alg on the NeHe page presumes this, though doesn't state it. See "What happens is that you go through every face, and if it is visible, ..."

Further the Z-pass technique on that page has some failure cases. First, you can't let any shadow volume intersect the near plane inside the frustum. If you do, shadow artifacts results. Think through it and you'll see why. The near plane clips away shadow volume pixels the eye can see, messing up the in-shadow/out-of-shadow count in the stencil buffer. You can generate near caps to try and avoid this, or you can use Z-fail stencil shadows in this case. This flips your near-clipping problem into a far clipping problem, and there are better options for dealing with that problem. While you can generate far caps, you also have the option of using "depth clamping" to have the GPU generate them automagically, or you can use an infinite far clip projection matrix which eliminates the possibility of far plane clipping.

Some silhouette determination techniques only work for convex casters. Look through how NeHe's doing that carefully and make sure it works for concave casters. Reading this may be useful:

* Fast silhouette detection for shadow volumes (OpenGL.org post) (http://www.opengl.org/discussion_boards/ubbthreads.php?ubb=showflat&Number=141852#Post1418 52)


And thinking of how shadow volumes work (extrusions of the silhouette), I don't see why these artifacts should be there.

When you determine your silhouette edges, draw the scene from the light's perspective and draw the silhouette edges in red. That may give you some insight. Failing that, draw your shadow volume tris in red wireframe, and rotate around the object to see if it looks good. I suspect you'll see cracks or missing faces.

Also, try punting your existing casters and use some very simple convex surfaces to verify that your alg works in the base case: cube, etc. Then try a really simple shape with a concavity.

aviel
05-13-2011, 08:46 PM
So, with a cube it was fine, but the problems begin to arise with an icosphere. All faces not facing the light (mostly, all faces beyond the silhouette) are being shadowed. I don't want that. How to prevent?

I'll show you what I mean: http://www.hereswhereyourewrong.net/sphere.png

And with a complex model:
http://www.hereswhereyourewrong.net/skelly.png

Dark Photon
05-14-2011, 10:09 AM
All faces not facing the light (mostly, all faces beyond the silhouette) are being shadowed. I don't want that. How to prevent?
I don't understand. With direct lighting on volumetric surfaces, all surfaces not facing the light "are" in shadow.

I think maybe one problem here is that you aren't doing lighting, or aren't doing it per-pixel, or aren't doing it correctly. Near the edges of your shadows, the surface normal is orthogonal to light rays, so dot(N,L) == 0, which means the diffuse lighting there is 0. So basically, at least as far as diffuse is concerned, the component of your lighting that gets shadowed should already have faded out to 0 at the point where your shadows start (i.e. where object turns back-facing in light space). So your light back-faces won't be so blatently obvious at at all. This will make your image look tons better, especially if you do fragment lighting or have enough verts so that vertex lighting doesn't look so bad. Also make sure your normals are normals for the surface being approximated, not normals for individual triangles.

Also, keep in mind you only shadow diffuse and specular, not ambient or emission. I'd just use ambient and diffuse for now to keep life simple.

This should get your object lighting and self-shadows looking much better and not so faceted and slightly confusing as they do now.

aviel
05-14-2011, 01:47 PM
I am just using ambient and diffuse and I'm only shadowing from the diffuse lighting.

The sphere rather explains the problem. You see how all of the sphere's faces beyond the silhouette are put into shadow? That shouldn't happen: the sphere's faces are lit properly if I don't shadow them. A face shouldn't shadow itself.

Dark Photon
05-14-2011, 01:54 PM
The sphere rather explains the problem. You see how all of the sphere's faces beyond the silhouette are put into shadow?
I don't understand this statement. For a convex object like this, all the faces completely beyond the silhouette (from the perspective of the light) are back-facing and thus not directly lit. They happen to be in shadow, but it's totally irrelevant that they are because the diffuse lighting contribution there is 0 -- so there's no diffuse light to shadow!

I guess I don't completely understand that sphere picture, or exactly what you're saying is wrong with it. It does look like it's chopped off in a very strange way. But without good lighting or a light position or anything, it's hard to tell what we're looking at.

aviel
05-14-2011, 01:57 PM
The reason it looks chopped off is because the shadow volume is affecting the back faces of the sphere when it shouldn't be. That's what I meant when I say that those faces shouldn't be in shadow: they're shadowing themselves and the shadow volume is incorrectly shadowing them.

jli--
05-14-2011, 07:15 PM
I suggest to not use stencil shadows, and to use some kind of shadow mapping. This is a really good tutorial on Variance Shadow Mapping: http://http.developer.nvidia.com/GPUGems3/gpugems3_ch08.html

aviel
05-14-2011, 07:29 PM
son of a bitch, soft edges? I hope this doesn't eat up too many resources.

/le sigh. time to redo shadowing code.

aviel
05-14-2011, 09:58 PM
I implemented a cheap hack. Objects that cast shadows don't receive them and objects that don't cast shadows do receive them. (Scenery receives shadows, doesn't cast, characters and such cast shadows, don't receive)