Shadow volume / wedge problem

Hello all,

I’ve been implementing the “Shadow wedge”-algorithm (original paper: http://www.cs.lth.se/home/Tomas_Akenine_Moller/pubs/soft_sig2003.pdf ) for quite some time and got really good results. I’ve used a floating point render target as the Li-Buffer for the hard shadow-pass and also for the wedge-pass.

My problem is how to render the hard shadows efficiently and run on ATI and nVidia cards (so NV_copy_depth_to_color is not an option). What I’ve done so far is sort of simulating the stencil buffer with a fp16 color buffer:

Hardshadow-pass (alternative #1):

-glEnable(GL_CULL_FACE), glFrontFace( GL_CW )
-set blending to additive (glBlendFunc(GL_ONE, GL_ONE))
-set fragment shader constant to +1 (this is returned in result.color and works like glStencilOp(GL_INCR_WRAP_EXT))
-render all shadow volume faces
-glFrontFace( GL_CCW )
-set fragment shader constant to -1 (this is returned in result.color and works like glStencilOp(GL_DECR_WRAP_EXT))
-render all shadow volume faces

This works, but I have to render the shadow volumes twice (which means transform vertices twice). I’ve come up with a different implementation which simulates GL_STENCIL_TEST_TWO_SIDE_EXT:

Hardshadow-pass (alternative #2):

-glDisable(GL_CULL_FACE)
-set blending to additive (glBlendFunc(GL_ONE, GL_ONE))
-provide normals for the shadow volume faces
-use a vertex shader that computes vertexNormal_dot_eyeVector and write +1 or -1 to result.color (should behave like glStencilOp( GL_INCR_WRAP_EXT / GL_DECR_WRAP_EXT ))
-render all shadow volume faces

Here’s my problem: Alternative #2 is not working correctly. Here you can see what I’m talking about:

Link to the pictures.

I’m providing normals for the shadow volume faces and I’m doing this in the vertex shader:

 !!ARBvp1.0

ATTRIB	v0     = vertex.position;

PARAM	WVP[4] = { state.matrix.mvp };
PARAM	eye    = program.local[0];
PARAM   svrt   = program.local[1];
PARAM	params = {  0, 1, -2, 0 };

#############################################
# output position

DP4		result.position.x, WVP[0], v0;		
DP4		result.position.y, WVP[1], v0;
DP4		result.position.z, WVP[2], v0;
DP4		result.position.w, WVP[3], v0;

#############################################
# calc if backface
TEMP	res, eyeVec, temp;

SUB		eyeVec, eye, v0;					# calc eyeVec

DP4		res.x, eyeVec, eyeVec;				# Normalize eyeVec (not really neccessary)
RSQ		res.y, res.x;
MUL		eyeVec,  eyeVec, res.y;

DP4		res.x, vertex.normal, eyeVec;		# check if front- or backfacing

MUL		res.x, res.x, svrt.x;				# svrt is Z_FAIL ? multiply by -1, else multiply by 1

SLT		res.y, res.x, params.x;				# res < 0.0 (Frontfacing means subtractive!) ? res.y = 1.0 : res.y = 0.0
MUL		res.x, res.y, params.z;				# Frontfacing? res.x = -2.0 : res.x = 0.0;
ADD		res.z, params.y, res.x;				# Frontfacing? res.z = -1.0 : res.z = 1.0;

MOV		result.color, res.z;				# output -1 or 1 for color

END
 

The fragment shader is just passing through the colors from the vertex shader.

What I don’t really get is:
I’m only writing -1 or +1 as output color.

Why am I getting fractional values?

Why is the same (like writing -1 or +1 depending on winding) working in the twopass version?

Any help will be appreciated,

GuentherKrass