I send in the vertices to tessellator (see image) using gluTessVertex().
As you can see it’s a “quad” with an indent. I have registered my own GLU_TESS_VERTEX callback, tessVertexCB(). If I understand correctly this function will be called for each vertex, in the order of each triangle, i.e. the three first calls to this function will give me the first triangle, the next three calls will give me the second triangle etc… Am I wrong?
These are the triangles I am getting (by index as in image):
1st. 1, 4, 5
2nd. 4, 1, 3
Already here the second triangle is wrong, i was expecting 2 as the last point in the second triangle. If vertex nr 4 is farther to the right the triangle 4, 1, 3 will intersect the 5, 4 line. Have I misunderstood how this works?
The tessellation output is not clear for me either but I guess registering the GLU_TESS_VERTEX callback is not enough. As far as I remember the GLU tessellator provides optimal sequence of primitives. The primitives might be triangles but also triangle fans or strips. In order to track the the primitive type you must also register GLU_TESS_BEGIN and GLU_TESS_END callbacks.
I found the problem. I was sending in 2D points, converting them back to 3D after the tessellation. The only problem was that for each point I sent in pointers to a list with 2D points, causing the Z value to become the X value of the next point.
facepalm
Hopefully some other newbie will read this and not have to post…