Initially, my code used the trackball code found in a gtkglextmm example. That code has been copied from the book "OpenGL Programming for the X Windows System". Since then I've cooked up some classes for the vectors, matricies and quaternions. As far as I can tell, my code and the original work identically. Drag the mouse around and rotation occurs. So what's the problem?
In the gtkglextmm example the model is a cube with one plane per cube face. Each plane is slightly offset from its cooresponding cube face. Offset from each plane is one of the following letters: 'G', 'T' or 'K'. If I place the mouse near the center of the cube and move the mouse in small circles around the center it causes the cube to rotate around the point I'm circling with the mouse. After just one circle with the mouse the letter will be on its side. Another circle and it's upside down. My code faithfully reproduces this effect.
It looks like all three axes are being rotated. I'd much rather have just 2 axes rotating, ie I move the mouse left/right, I get left/right rotation; I move the mouse up/down, I get up/down rotation. What causes the three axis behaviour? How can I get rid of it? Are there different implementations of the trackball code?