Happy new year everyone.
I’m trying to do stencil shadow volumes, and the shadow part is working fine. However you can see my problem from the picture below:
There used to be more artifacts on the side facing the light, but that was fixed with polygon offset. Anyone have any idea what can I do about this?
Your problem comes from not lighting the sphere properly.
How do you apply shadows to the scene? Do you just darken these places that receive shadow? If yes, then it’s completely wrong.
Describe what rendering passes are you using in your scene (source code will not be necessary). Typicaly it should be:
I’m doing the same way you described, but I think I know what the problem is. The polygons that are back-facing the light is in their own shadow, which is a special case that shouldn’t be shadowed at all, and instead let the lighting take over.
I’m gonna try and fix that, hopfully that was the problem.
hey if you find a solution, would you mind sharing it with us? i’ve been trying for quite some time to get rid of such artifacts without success 
Originally posted by GamerYZ:
The polygons that are back-facing the light is in their own shadow, which is a special case that shouldn’t be shadowed at all, and instead let the lighting take over.
If the lighting contribution for that light is added to existing image and if it does not contain any ambient term, the polygons fully back-facing the light should not receive any contribution from that light regardless of if they are in the shadow or not.
I’m doing the same way you described
Then why does your lighting affect polygons facing away from light source?
They should receive no lighting from that light source and since shadowed areas receive no lighting, too, then you won’t see the difference between black color (facing away = no lighting) and black color (inside shadow = no lighting).
So it seems like my first guess was right:
Your problem comes from not lighting the sphere properly.
And this is exactly what Komat pointed out, too. You should’nt add ambient lighting in passes done for particular light source - only diffuse and specular from that light source. You only do ambient lighting in pass 1 (before casting shadows).
Problem solved, thanks for the inputs everyone.
However I am 99.9% sure that I am doing ambient pass, stencil pass, then finally light pass without ambient (set to zero). Is that the correct way?
Anyways, for now I solved the problem by rendering the shadow caster twice for the final pass. I created 2 different fragment shaders, one shades all the fragment facing the light like normal, and the other shades all the fragments facing away from the light, but only shadow it if the fragment is under 2 or more shadows (1 being itself).
then finally light pass without ambient
and the other shades all the fragments facing away from the light
So what type of lighting do your polygons facing away from light receive? They should receive none and that is what I meant in my first post when I said you’re not lighting the sphere properly
If a polygon is facing away from light then it’s most certainly inside a shadow volume so it should receive no lighting.
The only artifact you can get is if your object has low number of polygons and polygon facing away from light will receive some lighting becaue it shares normal vector with some other polygon facing towards the light source. From what I see, your sphere’s tesselation is good enough to avoid that.
I actully found that solution from Nvidia site, except I moved the algorithm into shaders. They called it non-popping self-shadowing.
I understand what you are saying, that was my initial approch also. I think the problem was not improper lighting, but rather improper shadowing (is that the same thing? 
)
Hi GamerYZ,
Looking at the image, it does look like both improper lighting and improper shadowing.
The improper lighting is because the self-shadowed part doesn’t appear to have any ambient.
The improper shadowing part is intrinsic to the mismatch between the geometric shadowing and fragment shading.
Thanks -
Cass
I actully found that solution from Nvidia site, except I moved the algorithm into shaders.
how did you do it?
paper explaining the algorithm
http://developer.nvidia.com/object/robust_shadow_volumes.html
the demo + source code:
http://download.nvidia.com/developer/SDK/Individual_Samples/3dgraphics_samples.html
The demo had alot of extra options, the only useful insights for me were from the “display”, “draw_room”, and “draw_model” functions.
The first response was correct, your ambient term was non zero and you were shadowing it. You can have one or the other but not both.
The shadow at the boundary is inherently subject to z fighting polygonization and popping. Cass did have an interesting solution to popping if you can get it working, but it’s a second order problem and not the one dominating your image and the fudge requires a lot of dynamic geometry editing for the return.
I doubt your sphere will show significant popping issues with corrected lighting, and it is certainly not the problem you need to tackle first. All approaches will show problems if you shadow your ambient term in occluded region because it breaks the lighting math and highlights the innately messy, polygonized dynamic silhouette edge.
The check to draw polygon facing or not facing the light?
I calculated the dot product of the normal of the vertex and the vector to light in the vertex shader. Based on this result (<= or >= 0), I either discard or set color in the fragment shader.