glut and mouse wheel gives segmentation fault

[IndianRopeMan]

Hi,

i already posted this ineffectively to comp.graphics.api.opengl and on this board (but in the wrong section, they kindly directed me here).

I've got a strange problem that solely occurs
in my OpenGL app using glut (v3.7):
Whenever i scroll the mouse wheel forwards,
the program exits giving a segmentation fault.
When i scroll the wheel backwards, nothing happens. I have two subwindows and the mouse functions for each of them work fine, also the three buttons work as expected.
Libraries i use: X11, Xi ,Xm, glut, GL and GLU.
I'm on redhat9 using a MS Intellimouse (optical,usb) and scrolling with the mouse wheel works fine otherwise (see the mouse section of my XF86Cconfig below).
Since i can only use an usb mouse on my laptop, i have not yet tried other mice.

Has anybody got an idea what's going on
or what i could do to solve this?

I've already looked at alternatives like glfw but hesitate before porting.

Regards and THANKS,
Paul

---------------------------------------------
Section "InputDevice"
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "IMPS/2"
Option "Device" "/dev/input/mice"
Option "ZAxisMapping" "4 5"
Option "Emulate3Buttons" "no"
EndSection
---------------------------------------------

[FermatSpiral]

Very loosely I'd say it's an "off by one error". Have you tried to debug the app, watching the values returned by the mouse button pressed?

[IndianRopeMan]

Hi,

i didn't debug yet, thanks for your suggestion. I'll have to look how to do it.
I'm not quite sure what you mean with 'off by one': a unique error only occuring in my app?

Best regards

[FermatSpiral]

"Off by one" is a "more common than it should" programming error where somebody created a memory buffer, say, from 0 to 9 - but the actual code slips and thinks it is from 1 to 10, bites at the end of the buffer and dies with a segfault.

[ssross]

I think that I have found the problem. The scroll wheel sends mouse buttons 4 and 5, but an array of menu ids in the window structure only allowed for a 3 button mouse. Scrolling forward in a subwindow will effectively try to use the window's parent pointer as a menu id. Scroll back would try to use a child window pointer if any.

I sent a bug report and we'll see what official fix comes of it. For a quick hack (if you don't mind recompiling glut) you can either:

- try changing the '#define GLUT_MAX_MENUS 3' in glutint.h to 5
- validate the index of menu[] in glut_event.c line 425 (menuNum = window->menu[event.xbutton.button - 1] http://www.opengl.org/discussion_boards/ubb/wink.gif

I did the second and it 'seems' to work.

enjoy
scott