preventing self-shadowing

[Vexator]

i'm feeling a bit stupid.. i have z-fail shadow volumes which are working fine aside from the well-known glitches with self-shadowing on spheres etc. so i'd like to add an option which disables self-shadowing. i thought i could do so simply by rendering the extruded silhouettes as usual and then render the mesh itself in the stencil buffer.. but with which stencil op/func do i have to render the mesh?!

glStencilFunc( GL_ALWAYS, 0, ~0 );
 
// render silhouette pass 1
glCullFace( GL_FRONT );
glStencilOp( GL_KEEP, GL_INCR, GL_KEEP );
pkGroup->RenderSilhouette();
 
// render silhouette pass 2				
glCullFace( GL_BACK );
glStencilOp( GL_KEEP, GL_DECR, GL_KEEP );
pkGroup->RenderSilhouette();
 
// which stencil op/func changes do i need here?!
 
// render mesh itself
pkGroup->Render();

thanks in advance

[Humus]

GL_ZERO ought to do the trick, or GL_REPLACE with stencilRef as zero.

[Vexator]

thanks!
like this?

glStencilOp( GL_KEEP, GL_ZERO, GL_KEEP );
pkGroup->Render();

at first it seemed to work fine, but look what happens if i move the character below the floor:



i have to find a way to make it only do this if the mesh is not occluded i guess?

[Komat]

One way that might work with one limitation:

• Render shadow volumes for everything except for objects that should be not self-shadowed.
• Render the objects that should be not self-shadowed with lighting from that light.
• Without clearing the stencil render shadow volumes for the objects that were skipped in first step.
• Render rest of the lit geometry.

Obviously the problem is that objects that are not self-shadowed will not cast shadows on other such objects. I think that this is what Doom 3 does, if I correctly understand talk made by Carmack on one Quakecon.

[Humus]

Originally posted by Vexator:
like this?

No. Like this:

glStencilOp( GL_KEEP, GL_KEEP, GL_ZERO );

[Vexator]

this does not do anything thanks, though!

[Humus]

Could you list your rendering passes and the stencil calls for each of them?

[Vexator]

void World::RenderLighting()
{
	// get active camera
	Camera *pkCamera = Camera::GetActiveCamera();
 
	// render one pass for each light source
	for( unsigned int a = 0; a < m_vpkVisibleLightsSet.size(); a++ )
	{
		//get light
		Light *pkLight = m_vpkVisibleLightsSet[a];
 
		// enable stencil test
		glEnable( GL_STENCIL_TEST );
		Core::Get()->GetRenderer()->ClearFramebuffer( Renderer::BUFFER_STENCIL );
 
		if( pkLight->GetCastShadows() )
		{
			// setup states
			glStencilFunc( GL_ALWAYS, 0, ~0 );
			glColorMask( false, false, false, false );
 
			// for all models..
			for( unsigned int i = 0; i < pkLight->m_vpkShadowCastingGroupsSet.size(); i++ )
			{
				// activate extrusion shader
				Core::Get()->GetRenderer()->ActivateShader( m_pkExtrusionShader );
 
				// get current group
				Group *pkGroup = pkLight->m_vpkShadowCastingGroupsSet[i];
 
				// get light to model gl_Vertex
				Vector3D kLightVector = pkLight->GetTranslation()-pkGroup->GetParent()->GetTranslation();
 
				// re-compute silhouette
				pkGroup->ComputeSilhouette( kLightVector );
 
				// z-fail algorithm
				glCullFace( GL_FRONT );
				glStencilOp( GL_KEEP, GL_INCR, GL_KEEP );
				pkGroup->RenderSilhouette();
 
				glCullFace( GL_BACK );
				glStencilOp( GL_KEEP, GL_DECR, GL_KEEP );
				pkGroup->RenderSilhouette();
 
				// ------- HACK: try to prevent self-shadowing -------
 
				glStencilOp( GL_KEEP, GL_KEEP, GL_ZERO );
 
				// push matrix
				glPushMatrix();
 
				// apply model's view matrix
				Core::Get()->GetRenderer()->UploadMatrix( pkGroup->GetParent()->GetViewMatrix(), false );
 
				// render group (unshaded)
				pkGroup->Render( 0 );
 
				// Pop matrix
				glPopMatrix();
 
				// -----------------------------------------------------------
			}
 
			// deactivate shaders
			Core::Get()->GetRenderer()->DeactivateShaders();
 
			// reset states
			glStencilFunc( GL_EQUAL, 0, ~0) ;
			glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
			glColorMask( true, true, true, true );
		}
 
		// enable blending
		glEnable( GL_BLEND );
		glBlendFunc( GL_ONE, GL_ONE );
 
		for( unsigned int i = 0; i < pkLight->m_vpkIlluminatedGroupsSet.size(); i++ )
		{
			// get current group..
			Group *pkGroup = pkLight->m_vpkIlluminatedGroupsSet[i];
 
			// push matrix
			glPushMatrix();
 
			// apply model's view matrix
			Core::Get()->GetRenderer()->UploadMatrix( pkGroup->GetParent()->GetViewMatrix(), false );
 
			// check for custom shader, otherwise use to default lighting shader
			Shader *pkShader = pkGroup->GetCustomShader() ? pkGroup->GetCustomShader() : m_pkLightingShader;
 
			// activate shader
			Core::Get()->GetRenderer()->ActivateShader( pkShader );
 
			// render it
			pkGroup->Render( pkShader );
 
			// deactivate shaders
			Core::Get()->GetRenderer()->DeactivateShaders();
 
			// Pop matrix
			glPopMatrix();
		}
 
		// disable blending
		glDisable( GL_BLEND );
 
		// disable stencil test
		glDisable( GL_STENCIL_TEST );
	}
}

[Komat]

The anti self-shadowing render needs to be done using glDepthFunc( GL_LEQUAL ). Otherwise the depth test fails because the object geometry is already contained within the depth buffer with the same depth values.

One additional note. The glStencilOp( GL_KEEP, GL_KEEP, GL_ZERO ) render will erase all shadows on corresponding object that are casted by objects whose shadow volumes were rendered before so you might wish to introduce some ordering into the rendering of shadow volumes.

[Vexator]

The anti self-shadowing render needs to be done using glDepthFunc( GL_LEQUAL )

why have i been switching the depth function at all? Oo removed those switched and set GL_LEQUAL in the ctr, still does not help.. thanks anyway