When you create the new vertex in the gluCombie callback function, does glu free it? Or do I have to free all the new vertexen my self, and if so, how do i know at what point glu doesnt need them anymore.
GLU tessellator is not responsible to create a memory allocation for new intersect vertex, nor free the memory. GLU tessellator only tells you about the coords of the intersected vertex and its neighbours when the contour is self-intersected.
Indeed, your application must allocate a memory to store the new vertex data (coords, colour, UVs…), and have to clean it up yourself after gluTessEndPolygon() is called.
Notice that the last parameter of the combine callback; GLdouble **outData. It is the pointer to the new vertex data, which is created by your application. And you send it back to the tessellator (That’s why it is pointer to pointer or double pointer.), so the tessellator can pass it to GLU vertex callback function to render.
You may find a code for self-intersecting polygon here: tessellation
thanks dude!