PDA

View Full Version : Overflow and picking



zediah
07-17-2004, 07:11 PM
currently using vb6 and opengl, and the way its picking things is acting strange. On my computer, if i run it in windowed mode it goes fine, no errors. When it goes into fullscreen, i get an 'overflow' error everytime i attempt to click on an object.
When i press 'debug' on the usual error message thing, its saying that "glRotatef sceneroty, 0, 1, 0" is the culprit, if i remove this, it just goes down to the next glrotate, if i remove that, the next and so on.
I'm currently using a geforcefx5200 128mb ram graphics card - i attempt to show off my program to friend who's running a geforce4mx480 (i think thats the number) 64mb ram and he can't run either modes, gets overflow error. Another of my friends is using a radeon 9600 pro and he can run it in both modes fine!

it might be in my select function...i have no idea whats causing it. anyway, here's my select function below


Public Function RetrieveObjectID(x As Single, y As Single, ByVal Width As GLsizei, ByVal Height As GLsizei, selectb As Integer) As Integer
Dim ObjectsFound As Integer
Dim ViewPortCoords(4) As GLint
Dim selectbuffer() As GLuint
Dim selectedObject As GLuint
Dim i As Integer
Dim lowestdepth As GLuint
Dim buffersize As Integer

ReDim selectbuffer(1000 * selectb)
buffersize = UBound(selectbuffer)
ObjectsFound = 0
glSelectBuffer buffersize, selectbuffer(0)
glGetIntegerv glgViewport, ViewPortCoords(0)
glRenderMode (GL_SELECT)
glInitNames
glPushName 0

glMatrixMode (GL_PROJECTION)
glPushMatrix
glLoadIdentity
gluPickMatrix x, ViewPortCoords(3) - y, 2, 2, ViewPortCoords(0)
gluPerspective 45#, Width / Height, 0.1, 100#

glMatrixMode (GL_MODELVIEW)
glLoadIdentity
glPushMatrix
DrawGLScene
glPopMatrix


glMatrixMode (GL_PROJECTION)
glPopMatrix
glFlush

glMatrixMode (GL_MODELVIEW)

ObjectsFound = glRenderMode(GL_RENDER)

If ObjectsFound < 0 Then
MsgBox "overflow!"
End If

If ObjectsFound > 0 Then
lowestdepth = selectbuffer(1)
selectedObject = selectbuffer(3)
For i = 1 To ObjectsFound
If (selectbuffer((i * 4) + 1) < lowestdepth) Then
lowestdepth = selectbuffer((i * 4) + 1)
selectedObject = selectbuffer((i * 4) + 3)
End If
Next i
End If

RetrieveObjectID = selectedObject
End Function