I have a problem with stencil shadows. I coded up an algorithm that calculates the silhouette of the projection (using the standard algorithm that checks the facing of two adjacent triangles in relation to the light position).
It really seems to work nicely, as the following screenshot shows:
Now when I use a different model, the shadow doesn't work any more
The question is: What goes wrong?
I already had different ideas that I tried:
1. The second model uses more than one mesh - this could be a problem as there will be edges the virtually are the same but are in different meshes. I already checked this, but this seems not to be the single problem - even if I only draw the shadow of one mesh, I get these ugly artefacts. I also merged all meshes together, but this doesn't help much, even when I collapse all identical vertices.
2. Another guess was some numeric problems when calculating the plane equations of the triangles. So I checked all normals and discarded the triangles with normals below a certian threshold.
3. I also suspected that I might have some problems with some edges that do not belong to two triangles (that is a hole in the model). I included a special case for these and made sure that they will always be included in the silhouette - but this didn't help much either.
So here is the question: What bad properties might the model have and how can I work around these problems? Both models have all triangles with correct orientations (ie backface culling really works) and they are both more or less in the same scale.
Any ideas are welcome.