every open source engine i looked at and which uses the stencil buffer to simulate shadows does it by masking out the area into which shadows have to be drawn, then they render a semi-transparernt black polygon covering the whole screen into the masked area.
but this is actually not realistic at all, as the masked area is still lit by the light source and the shadow is “added” afterwards.

so i try it another way. i mask out the area which should be in shadow, the i deactivate the light source casting the shadow and render the whole world once again, this time into the masked area. this is slower, but more realistic and also enables colored shadows (if there is global colored lighting e.g.) and specular highlights being rendered correctly.

this is how it looks then:

http://www.vexator.net/test.jpg

k now i have 2 questions ^^

1. one problem occurs if i want multiple light sources to cast shadows. how can i blend the individual shadows cast by different sources together? i cannot use blending as i had to disable lighting for this, right?

2. is there a way to speed up rendering the world multiple times? i have to render it for every light source 1 time, but i read something about locking/unlocking the buffers if the geometry is not being moved or sth like this…?! cannot find it anymore. i’m using vertex arrays and vbo’s for all my data if this helps

3. the stencil buffer can have 8bit… for what can i use them? so far i’m only using a 1bit stencil buffer.

thx in advance

Originally posted by Vexator:
[b]

1. one problem occurs if i want multiple light sources to cast shadows. how can i blend the individual shadows cast by different sources together? i cannot use blending as i had to disable lighting for this, right?
   [/b]
   One possible Solution looks like this:

• [li]Render your scene with ambient light only[/li] [li]Set glBlendFunc(GL_ONE, GL_ONE)[/li] [li]For every light source: Compute shadow volumes and render to stencil; Render scene with diffuse and specular lighting only

2) is there a way to speed up rendering the world multiple times? as i have to render it for every light source 1 time. but i read something about locking/unlocking the buffers if the geometry was not moved or so…?! cannot find it anymore. i’m using vertex arrays and vbo’s for all my data if this helps

Render to depth buffer in the ambient pass, then set glDepthFunc(GL_EQUAL). Add clip planes around your point lights and add another culling step that discards objects that are too far away from your light source. Merge multiple directional lights into a single pass.
Edit: This is still very fillrate intensive. It also interacts badly with semi-transparent objects.

#Render your scene with ambient light only

Set glBlendFunc(GL_ONE, GL_ONE)

For every light source: Compute shadow volumes and render to stencil; Render scene with diffuse and specular lighting only

but if i want to use glBlendFunc(…) then i have to enable blending… and that’s only possible if i disable lighting first - or i am wrong?

but thx so far

lighting has no effect on blending - having lighting enabled simply means the ‘color’ is computed from the light interactions (glLight/glMaterial) before being blended, whereas with lighting disabled the ‘color’ is specified directly by you (using glColor).

setting the blend function like you recommended results in totally white shadows Oo

Originally posted by Vexator:
so i try it another way. i mask out the area which should be in shadow, the i deactivate the light source casting the shadow and render the whole world once again, this time into the masked area. this is slower, but more realistic and also enables colored shadows (if there is global colored lighting e.g.) and specular highlights being rendered correctly.

If I understand you correctly you are drawing the shadow in way similiar to the following:

Draw scene lit by lights 1..X
Use stencil buffer to mark areas that should be in the shadow of light 1.
Draw scene lit by lights 2..X into shadow area.

The ordinary way this is done which is the one to which refered memfr0b is somewhat oposite.

Use stencil buffer to mark areas that should be in the shadow of light 1.
Draw scene lit by light 1 into area outside shadow of the light 1.
Use stencil buffer to mark areas that should be in the shadow of light 2.
Draw scene lit by light 2 into area outside shadow of the light 2 using additive blending.
....
Use stencil buffer to mark areas that should be in the shadow of light X.
Draw scene lit by light X into area outside shadow of the light X using additive blending.

ok so it seems that we talked at cross-purposes yes i do it the way you thought i would. so what would be the way to blend my shadows then? or would it be better to rewrite my code and do it the way memfr0b thought of? but that’d really be a pain in the ass thx so far!

Originally posted by Vexator:
setting the blend function like you recommended results in totally white shadows Oo
There is a two approach with shadows: darkening enlighted scene or lighting dark scene. First one is bad and easy to implement, second approach produce correct result but it is a bit difficult to implement.

You have to use offscreen buffers (FBO or pbuffer). Since current FBO implementation doesn’t support stencil buffers, I suggest you to use pbuffers with additional stencil and aux buffers. Let pbuffer’s color buffer contain (ambient + shadows) * texture term and pbuffer aux1 contain specular term.

If you want correct lighing and shadows do following:

• create pbuffer with stencil, depth, color and at least one aux buffer
• switch context to pbuffer
• clear everything (depth, color, aux)
• select color buffer as draw buffer
• render ambient & depth (ambient should be close to dark). You can use precalculated ambient term in vertex color or texture maps.

result is ambient enlighted scene

• disable depth write
• enable additive blending (gl_one, gl_one)
• for each light
  **** clear stencil
  **** apply shadow volume on stencil buffer
  **** enable stencil test
  **** render polygons affected by this light, but only diffuse term
  **** select aux1 buffer as draw buffer
  **** render polygons affected by this light, but only specular term
  **** select color buffer as draw buffer

result is scene with shadows in color buff and specular term in aux1 (also affected by shadows)

• disable stencil test
• set blending to modulate (gl_dst_color, gl_zero)
• render polygons with textures

result is scene with shadows and textures

• switch to main context
• render screen aligned quad with pbuffers color texture
• enable additive blending (gl_one, gl_one)
• render screen aligned quad with pbuffers aux1 texture (specular)

yooyo

uff… i have no experience with pixel and auxilary buffers yet but i understand what you want me to do. oh hell… seems like a whole bunch of work but thanks! i’ll give my best.

EDIT: if i have my shadows in a pixel buffer wouldn’t that also mean that i could soften them?

Think about writing renderer based on image compositing. Whole idea is to render several images and mix them all in final image using your pixel shader for compositing.

In your case you have to render following layers:

• depth layer
• ambient_layer(depend on depth layer)
• diffuse_layer(accumulated diffuse term from all light sources, affected by shadow & depend on depth layer).
• specular_layer(accumulated specular term from all light sources, affected by shadow & depend on depth layer)
• texture layer(depend on depth layer)
  And you can add more layers:
• self_illumination_layer (depend on depth layer & black fog)
• heat_layer (depend on depth layer)

Compose following layers:
L1 = blured(self_illumination). Blur is not too expensive on todays hw.
L2 = (ambient + diffuse)*texture + specular
L3 = L1 + L2
L4 = heat_layer

Final image should be:
Final = offset(L3, L4) -> read pixel from L4 and add to current mapping coordinate and use new coordinate to fetch pixel from L3.

Result is fully shadowed scene with blured self_illumination and all this is affected by heat.

You can optimize whole process by compositiong several layers in one layer using several passes, (just like in my previous post) or if you detect that hardware support multi render target, you can calcualte few layers in single rendering pass (ie. diffuse -> color buffer and specular -> aux1 in single pass)

Sharing depth buffer between one or more layers (aux buffers) can give you performance boost because of early-z rejection if you have expensive shaders.

yooyo

It’s not neccesarily that complex.

Just do the following:

1. render scene with all lights disabled
2. set ambient to black and blend function to GL_ONE, GL_ONE
3. set depth function to GL_EQUAL
   4a) render shadow volumes to stencil buffer
   4b) render scene with light enabled OUTSIDE the shadow region only

repeat steps 4 for every light source…

It should always give correct results. If there are areas that should be colored but they end up white, this means the light sources are too bright. If you have an object lit by multiple light sources, you should make them darker because otherwise you will overflow the color values (values above 1.0 are clamped).

oke for now i’ll try to solve it the way overmind proosed, i’m just not familiar enough with fbos and shading. but i’ll copy/paste your code yooyo, and will take a closer look when i think i can do it thx guys

yay it works great
i’ve been busy with job search the last weeks but now i finally find time to work on:

i have 3 new questions:

• i have to render the world once for each light source which slows down rendering a lot. i read somewhere (but lost my source) that you can speed up rendering in ogl if you have to render a scene multiple times and you don’t need to translate or rotate anything… it was said that the counterpart in d3d is lock() and unlock() or so?!
• vertex arrays also can hold boolean data called edge flags. are these supposed to hold the edge flags of stencil shadows? if yes, how can i use them to improve performance?
• can i detect somehow if a stencil mask is visible on the screen?

thx in advance