Vertex shader trouble

Hello, all. I’m trying to implement stencil shadows in a project I’m working on, using ARB vertex shaders, and I’ve run into some trouble. I’m following this article on Gamasutra, and it does not seem to be working.

Before rendering the shadow volume (both front and back faces), I enable my vertex program, which is as follows:

!!ARBvp1.0

ATTRIB vPos = vertex.position;
OUTPUT oPos = result.position;

PARAM lPos = program.env[0];

TEMP temp;

ADD temp.xyz, vPos, -lPos;
MAD temp, vPos.w, lPos, temp;
DP4 oPos.x, state.matrix.mvp.row[0], temp;
DP4 oPos.y, state.matrix.mvp.row[1], temp;
DP4 oPos.z, state.matrix.mvp.row[2], temp;
DP4 oPos.w, state.matrix.mvp.row[3], temp;

END

My understanding is that this program will extrude vertices with a zero W component, and do nothing to vertices with a one W component.

When rendering the shadow volume, I do:

glVertex4f (verts[1][0], verts[1][1], verts[1][2], 1);
glVertex4f (verts[0][0], verts[0][1], verts[0][2], 1);

glVertex4f (verts[0][0], verts[0][1], verts[0][2], 0);
glVertex4f (verts[1][0], verts[1][1], verts[1][2], 0);

Where the verts array holds the coordinates of the edge that is being extruded.

Before drawing these shadow volumes, I also do:

glProgramEnvParameter4fARB (GL_VERTEX_PROGRAM_ARB, 0, lightvec[0], lightvec[1], lightvec[2], 1.0f);

Where lightvec holds the location of the light I’m using to create the shadow volume.

Nothing actually shows up when using the vertex shader.

When I used the following code to calculate the extrusion of the points by hand, it worked. (I already did glVertex3f for the first two, unextruded points)

Vector d1, d2;

d1 = verts[0] - lightvec;
d2 = verts[1] - lightvec;

d1.Normalize ();
d2.Normalize ();

d1 = d1 * 10000;
d2 = d2 * 10000;

glVertex3fv (d1);
glVertex3fv (d2);

Any assistance here would be very much appreciated!

What fragment program or texturing or lighting are you using? You don’t emit any normal or texture coordinates or color, so things like alpha testing could kill fragments, for example.

I’d try blasting all possible state to known, sane values right where you enable the vertex program. Also note that it’s the responsibility of the vertex program to move to output color from input color if you’re using that; same thing for texture coordinates.

Quoth The Spec:

When in vertex program mode, all attributes of a transformed vertex are
undefined at each vertex program invocation. Any results, or even
individual components of results, that are not written to during vertex
program execution remain undefined.

I put a “MOV result.color, vertex.color;” instruction in there, still no go.

This is just drawing the shadow volume, and color writes are disabled anyway…

Alpha testing is also disabled.

Something I just noticed:

The first frame it draws, it seems to be showing a shadow, but it is not the correct shadow, and it is much smaller. It seems to shrink for the next 2 or 3 frames (running at 30+ FPS, it’s hard to tell ), and then it is just gone.

I Don’t know what that means, though…

Does your camera, your object or your light moves ?

Also, just something to try out : disable vertex program and render your infinite quad with that (use GL_TRIANGLES instead of GL_QUADS) :

glVertex4f (verts[1][0], verts[1][1], verts[1][2], 1);
glVertex4f (verts[0][0], verts[0][1], verts[0][2], 1);
glVertex4f (-lightvec[0], -lightvec[1], -lightvec[2], -1);

[This message has been edited by vincoof (edited 04-25-2003).]