View Full Version : gluTesselator Heap Corruption

03-16-2009, 11:02 PM
I've been searching around for a solution but am still lost. I've been trying the gluTesselator version " Microsoft Corporation" and have encountered a heap corruption problem after the end of gluTessEndPolygon. The tesselator is able to output the result of the tesselation fine, but once my program runs any code that does a heapchk, such as a new operator or a vector operation, it will crash.

I followed the red book quite closely for the initialization steps. Then I create the contour dynamically from user input and store the points in mCurrPolygon vector (so i just transfer it over to a new array for now):

// fill up temp with point data
GLdouble** temp = new GLdouble*[mCurrPolygon->size()];
float tx, ty;
for(int i=0; i<mCurrPolygon->size(); i++)
// basically copying data from mCurrPolygon to GLdouble array
mCurrPolygon->getPoint(i, tx,ty);
GLdouble* newPt = new GLdouble[3];
temp[i] = newPt;
newPt[0] = (double)tx;
newPt[1] = (double)ty;
newPt[2] = 0;

Following that, i use the tesselator:

gluTessBeginPolygon(mTesselatorLine, NULL);
for(int i=0; i<(int)mCurrPolygon.size(); i++)
gluTessVertex(mTesselatorLine, reinterpret_cast<GLdouble*>(temp[i]), reinterpret_cast<void*>(temp[i]));

The tesselator works fine and the callbacks show the appropriate result. I managed to print out the calls from begin, vertex and end perfectly.

However, when _heapchk is called above, the program usually says that there is "Corruption of the heap", or sometimes it says "Access violation writing location" at gluTessEndPolygon after the callback to endCallback is made.

I have also tried using a vector<GLdouble*> instead of a 2D array, but to no avail. Can anyone help? I am rather convinced there is some problem inside gluTessEndPolygon that is writing to an illegal location but I have not looked at the source code for glu.

03-17-2009, 12:12 AM
I found the solution. I was declaring callbacks like:

void tessLineEndCallback(void)

and setting them like so

gluTessCallback(mTesselatorLine, GLU_TESS_END,
(GLvoid (__stdcall *) ()) &amp;tessLineEndCallback);

but I should have been declaring the callbacks with __stdcall as well:

void __stdcall tessLineEndCallback(void)

I had to add __stdcall to the cast to make the code compile but I did not realise that I should have added it to the function as well. The difference between the __stdcall and __cdecl calls caused the heap corruption.