View Full Version : Shadowmap Halo Effect

04-03-2011, 04:28 PM
Hi guys. I've been playing with shadow maps in order to get hold of some nice looking soft shadows. I'd been asked to make something pretty much like the PCSS paper. I'd tried playing around with VSM but I couldnt quite get it to work how I wanted. In the end I tried to settle for PCF as that forms the basis for PCSS. The problem was I was having an awful lot of problems with the Moire pattern and the Z-Fighting on certain surfaces. It didnt look too pretty so I looked around. I found an example on Gamedev where a blur was applied and the texture was then merged with the scenery.

At the moment, I have a PCF solution that goes like this:

1) Render the Depth map to an FBO. This uses the 32bit depth buffer and nothing else. Z values go straight in as floats
2) Bind a second FBO and render the shadows using the PCF algorithm.
3) draw this texture to a screen aligned quad and apply a pingpong guassian blur.
4) render geometry normally to a third fbo.
5)Finally, draw another quad with the two textures bound to two texture units and combine with a shader

My first question is, can get away with 2 steps by using multiple render targets to render both the depth the the colour? I suspect I can. I'd just need one shader attached to one colour target and another shader attached to another.

The main issue I have though is a halo-ing effect:

If you look at the Red dragon, there is a fringe around it. If I dont blur the texture the fringe is still there but obviously, not continuous. I've posted a video of this as well:


Annoyingly it doesnt seem to work as well as I'd like. My second question is what could be causing it? I'd really like to get rid of it.

Finally, I understand that PCSS takes the depth into account and modifies the kernel size (the number of samples) adaptively, depending on the distance from the blocker? Is that about right? I get the feeling that shouldnt be too hard to implement over the top of PCF?

Here is my PCF Shader

uniform float pcfScale;
uniform int pcfSamples;
uniform int texMapSize;
uniform sampler2DShadow depthTexture;
uniform float bottomLine; // We really shouldnt need this but we do :(

varying vec3 N, V, L, M;
varying vec4 q;

vec2 rand(in vec2 coord) //generating random noise
float noiseX = (fract(sin(dot(coord ,vec2(12.9898,78.233))) * 43758.5453));
float noiseY = (fract(sin(dot(coord ,vec2(12.9898,78.233)*2.0)) * 43758.5453));
return vec2(noiseX,noiseY)*0.004;

float computeBasic() {
vec3 coord = 0.5 * (q.xyz / q.w + 1.0);
float qw = q.z/q.w;
if ( qw - shadow2D( depthTexture, coord ).r < bottomLine){
return 1.0;
return 0.0;

float computePCF() {
float sum = 0.0;
float x =0.0;
float y = 0.0;
float texscale = 1.0 / float(texMapSize);
vec3 coord = 0.5 * (q.xyz / q.w + 1.0);
float bottom = 0.5 - float(pcfSamples) / 2.0;
float top = -bottom;
float qw = q.z / q.w;

for (y = bottom; y <= top; y += 1.0){
for (x = bottom; x <= top; x += 1.0){
vec2 t = vec2(x,y);
t = t * texscale * pcfScale;
coord.x += t.x;
coord.y += t.y;
if (qw <= shadow2D(depthTexture, coord).r + bottomLine)
sum += 1.0;

return sum / float(pcfSamples * pcfSamples);

void main(void) {

float shadow = computePCF();
gl_FragColor = vec4(shadow,shadow,shadow,1.0);

And the combine step

uniform sampler2D shadowTexture;
uniform sampler2D baseTexture;

void main() {

float shadow = texture2D(shadowTexture, gl_TexCoord[0].st).r;
vec4 base = texture2D(baseTexture, gl_TexCoord[0].st);
gl_FragColor = base * shadow;


So far it seems ok. There will be other effects being combined with this so I suspect multiple targets will be the thing but I'm concerned about the halo effect. Annoyingly no matter how hard I tried I couldnt get rid of that damn moire pattern! >< It was proving quite tricky!

Dark Photon
04-03-2011, 05:02 PM
In the end I tried to settle for PCF as that forms the basis for PCSS. The problem was I was having an awful lot of problems with the Moire pattern and the Z-Fighting on certain surfaces.
Couple really easy things you can do:

1) Only render light-space back-faces into shadow maps, and
2) Only attenuate diffuse and specular with your shadow map lookups.

This gets rid of almost all of the acne/moire (assuming your shadow map texels aren't huge). And then to kill almost all the rest, apply a small bit of PolygonOffset when rendering the shadow map:

glPolygonOffset( 1.1, 4.0 );

Here's the insight: if you store light-space front-faces in the shadow map, you'll have the lit side (the bright side) of your object fighting with shadow map lookup/projection quantization artifacts (acne/moire/etc.). So don't do that. Cast the light-space back-faces into the shadow map. Then only your dim sides have to play that game. But for those faces, they only get ambient anyway, right? (diffuse and specular are 0 there). And you're only using the shadow term to attenuate diffuse and specular. So basically you don't see the acne/moire back there because you're attenuating 0 to 0. :cool:

With just the first two changes, you only have to fight the acne/moire thing for: 1) points/faces on your object tangent to light rays, 2) for areas on an object where the front and back faces are very close together (relative to the shadow map texel size), and (as a special-case of #2) 3) for 2D "impostor" type objects, where there is no thickness to the object. It's for these cases you end up doing PolygonOffset and/or adding some fancier filtering like EVSM.

04-04-2011, 01:09 AM
Thanks for that headsup. I've tried both of these and I found that culling te front face BUT then turning off all culling on the next pass worked quite well for things like the cubes and planes I was rendering. Polygon offset didnt work very well for me but yes, with the extra light term, you can get away with that.

Falloff, or attenuation is a BIG issue. Say you have falloff for shadows, ala VSM. This results in weird artefacts because if one object eclipses a second, the second object will be in shadow but its BACK face will take on the shadow of its front face and not be completely in shadow because its reading the occulder depth of the first object and not the second. this can *almost* be avoided with the lighting pass.

Still, I've no idea where these halos are coming from! :S