View Full Version : mouse picking in opengl using `gluUnProject`

06-06-2013, 08:40 AM
I have a scene that I am rendering couple of cubes in it with openGL (program structure is not using GLUT,it is in win32 program structure but I just draw cubes with `glutSolidCube`)
now I want to select these cubes by mouse by picking. this is what I am doing:
first when the user clicks the mouse button on the scene I get the mouse position and trying to find its coordinates in scene coordinates (templateSkeletons is a skeleton I created out of cubes, nothing more):

if (mouse.buttonPressed(Mouse::BUTTON_LEFT))
templateSkeletons[0].selectionMode = true;
Vector3* points;
points = GetOGLPos();
templateSkeletons[0].selectionMode = false;
this is the `GerOGLPos` function that I am retrieving the coordinates in the scene (notice that I have my own camrea and its own projection matrix but I am fetching projection matrix here in this function just by calling `glGetDoublev (GL_PROJECTION_MATRIX, projmatrix);` is this wrong and I should get my own camrea's projection matrix ? ) :

Vector3* GetOGLPos()
{Vector3 pointsOnLine[2];
double mvmatrix[16];
double projmatrix[16];
int viewport[4];
double dX, dY, dZ, dClickY,zz;
glGetIntegerv(GL_VIEWPORT, viewport);
glGetDoublev (GL_MODELVIEW_MATRIX, mvmatrix);
glGetDoublev (GL_PROJECTION_MATRIX, projmatrix);
dClickY = double (viewport[3] - mouse.yPos());
// OpenGL renders with (0,0) on bottom, mouse reports with (0,0) on top
//glReadPixels( mouse.xPos(), int(dClickY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &zz );
gluUnProject ((double) mouse.xPos(), dClickY, 0.0, mvmatrix, projmatrix, viewport, &dX, &dY, &dZ);
pointsOnLine[0] = Vector3( (float) dX, (float) dY, (float) dZ );
gluUnProject ((double) mouse.xPos(), dClickY, 1.0, mvmatrix, projmatrix, viewport, &dX, &dY, &dZ);
pointsOnLine[1] = Vector3( (float) dX, (float) dY, (float) dZ );

return pointsOnLine;
now I suppose I have two points indicating my picking ray in the scene. now when I render the cubes I try to calculate the distance of the line created by the ray and the cube, and if it is smaller than a value I change the color of the cube to know that I picked it (jointsOfSkeleton indicates each cube that is created the skeleton nothing more, in here I am testing just the cube number 6 in array ):


distToLine = Vector3::PointToLineDistance3D(rayPoints[0],rayPoints[1],Vector3::Vector3(jointsOfSkeleton[6].x,
//distToLine = sqrt(distToLine);
if(distToLine < 0.5)


when I click on irrelevant positions on the window I see the colors of cube changes, it doesn't work right, I am watching the distances on the debugger and the distances doesn't look right. and this is function I used for finding Line-point distance:

static float PointToLineDistance3D(Vector3 a, Vector3 b, Vector3 point)

Vector3 lineDirection = b - a;
float t = (Vector3::dot(point,lineDirection) - Vector3::dot(lineDirection,a))/(Vector3::dot(lineDirection,lineDirection));
Vector3 direction;
direction.x = a.x + (lineDirection.x *t) - point.x;
direction.y = a.y + (lineDirection.y *t) - point.y;
direction.z = a.z + (lineDirection.z *t) - point.z;

float ShortestDistance = sqrtf((direction.x*direction.x)+(direction.y*direc tion.y)+(direction.z*direction.z));

return ShortestDistance;


sorry for long post but I will appreciate your patience to read and helps and insights. where I did wrong.

06-08-2013, 03:20 PM
I changed the PointToLineDistance3D function, now it works well. but the problem now is that I have my own camera, so I load its projection and modelview matrices when I am doing glUnProject.
as you see in the snapshot I have selection when I am not on joints the joint gets selected. it is like that my ray come from another point and it does not match with the cursor
any ideas ?