debugging crashes in the DLL

Sometimes my app is crashing when I make a certain change such as changing the texture that is suppose to be applied to an object.

I really dont see how this could cause a crash. It points to a certain line in the vendors DLL (ATI for now)
In the dissassembly window, I see

rep mov [edi], [esi]

or something like that where the address in esi is probably wrong.

I placed some code to try to catch the problem. It basically checks pointers if they have a certain address that esi has, but it’s not crashing anymore.

Is there anything I can do to catch these problems before they happen? The dissassembly doesnt tell me where the problem lies, or how it’s happening.

I consider my app to be well programmed, so I dont think Im going beyond array boundaries.

PS: Im using plain glDrawRangeElements. NO VBO or display list or …

Help!

[This message has been edited by V-man (edited 11-04-2003).]

You can manually unwind the stack by inspecting the stack frames in memory, and get a return address into your code. Then you can set EBP and ESP correctly to return to your code, and execute a RET instruction in the debugger, to end up pointing at the place that called into GL. At that point, you can examine the state of your own code, with symbols and a stack trace.

Oh boy, crashes in the driver are not nice as they are typically running on a different thread and you cannot get a crash context from the main program.

What I did to solve a crash of mine was to run a modified version of glTrace on my app and view the OpenGL calls just before a crash. Even tho it crash in a slightly different place each time, it always crashed on the glDrawElements call. (these particular calls were with dynamic system mem geometry - all VBO stuff done in other calls did not crash)

Then I ran some code before the GL call to make sure the vertices and indices were valid. (Which they were) However, I did find that if I padded my index array with an extra ~40 bytes at the end, the crash no-longer occured.

I have no idea why this was the case, I can only assume the driver was accessing beyond the index array end in some copy call and caused some sort of memory error.

BTW: This was an ATI 9700 with cat 3.6/3.7 drivers. Have not tested to see if it was fixed in 3.8.

You may want to try the above “HACK” and see what happens.

Originally posted by jwatte:
You can manually unwind the stack by inspecting the stack frames in memory, and get a return address into your code. Then you can set EBP and ESP correctly to return to your code, and execute a RET instruction in the debugger, to end up pointing at the place that called into GL. At that point, you can examine the state of your own code, with symbols and a stack trace.

Im not sure how to do that. When it crashes, it gives 0xC0000005 access fault. I thought it was not possible to change register values after a crash. I beleive I tried something of the sort in the past.

>>Set EBP and ESP correctly<<<

That’s a problem!

>>>What I did to solve a crash of mine was to run a modified version of glTrace <<<

You dont absolutly need glTrace I think. You can just have a function that validates your arrays.

I added a check for seeing if the right client arrays are enabled/disabled. I’ve had that nasty in the past.

I have to keep trying I guess.

OK, it’s solved.
Looks like the problem was that I forgot to put a parenthesis somewhere, and the wrong address was beeing used for glTexCoordPointer

PS : I wish their was a way to move backwards into the code. Like an undo operation. That would be a must have for a debugger. Anyone know of any such solutions?

[This message has been edited by V-man (edited 11-06-2003).]

V-man: there are In-Circuit Emulators that can do that, more or less. However, they’re expensive, and slow.

Regarding crashes in the driver being in a different context, that’s usually true for DirectX, as there is a stack switch when you switch into kernel mode, and DirectX switches to the kernel a lot. However, in OpenGL, you’re usually still in the OpenGL DLL which lives in your user address space, and the stack is quite recoverable.

Almost all calls in OpenGL use synchronous copy semantics, i e you can free() your memory once you’re done calling TexSubImage() or DrawElements(), so having the driver reference your pointers in another thread would be in wild violation of the spec; I really don’t think that’s the case in most OpenGL crashes.

Again, you need to find a good return address on the stack by examining memory, set ESP to point to that return address, find a RET instruction, do a “set next instruction” and step one instruction, to get back to your application and see where you crashed. This works just fine in most debuggers (WinDbg, MSDEV, etc)