I have a scene that I am rendering couple of cubes in it with openGL (program structure is not using GLUT,it is in win32 program structure but I just draw cubes with `glutSolidCube`)

now I want to select these cubes by mouse by picking. this is what I am doing:

first when the user clicks the mouse button on the scene I get the mouse position and trying to find its coordinates in scene coordinates (templateSkeletons is a skeleton I created out of cubes, nothing more):

if (mouse.buttonPressed(Mouse::BUTTON_LEFT))

{

mouse.update();

templateSkeletons[0].selectionMode = true;

Vector3* points;

points = GetOGLPos();

templateSkeletons[0].setIntersectionPoints(points[0],points[1]);

}else

templateSkeletons[0].selectionMode = false;

this is the `GerOGLPos` function that I am retrieving the coordinates in the scene (notice that I have my own camrea and its own projection matrix but I am fetching projection matrix here in this function just by calling `glGetDoublev (GL_PROJECTION_MATRIX, projmatrix);` is this wrong and I should get my own camrea's projection matrix ? ) :

Vector3* GetOGLPos()

{Vector3 pointsOnLine[2];

double mvmatrix[16];

double projmatrix[16];

int viewport[4];

double dX, dY, dZ, dClickY,zz;

glGetIntegerv(GL_VIEWPORT, viewport);

glGetDoublev (GL_MODELVIEW_MATRIX, mvmatrix);

glGetDoublev (GL_PROJECTION_MATRIX, projmatrix);

dClickY = double (viewport[3] - mouse.yPos());

// OpenGL renders with (0,0) on bottom, mouse reports with (0,0) on top

//glReadPixels( mouse.xPos(), int(dClickY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &zz );

gluUnProject ((double) mouse.xPos(), dClickY, 0.0, mvmatrix, projmatrix, viewport, &dX, &dY, &dZ);

pointsOnLine[0] = Vector3( (float) dX, (float) dY, (float) dZ );

gluUnProject ((double) mouse.xPos(), dClickY, 1.0, mvmatrix, projmatrix, viewport, &dX, &dY, &dZ);

pointsOnLine[1] = Vector3( (float) dX, (float) dY, (float) dZ );

return pointsOnLine;

}

now I suppose I have two points indicating my picking ray in the scene. now when I render the cubes I try to calculate the distance of the line created by the ray and the cube, and if it is smaller than a value I change the color of the cube to know that I picked it (jointsOfSkeleton indicates each cube that is created the skeleton nothing more, in here I am testing just the cube number 6 in array ):

if(selectionMode)

{

distToLine = Vector3::PointToLineDistance3D(rayPoints[0],rayPoints[1],Vector3::Vector3(jointsOfSkeleton[6].x,

jointsOfSkeleton[6].y,jointsOfSkeleton[6].z));

//distToLine = sqrt(distToLine);

if(distToLine < 0.5)

glColor3f(1.0,0.0,0.0);

else

glColor3f(1.0,1.0,1.0);

}

when I click on irrelevant positions on the window I see the colors of cube changes, it doesn't work right, I am watching the distances on the debugger and the distances doesn't look right. and this is function I used for finding Line-point distance:

static float PointToLineDistance3D(Vector3 a, Vector3 b, Vector3 point)

{

Vector3 lineDirection = b - a;

float t = (Vector3::dot(point,lineDirection) - Vector3::dot(lineDirection,a))/(Vector3::dot(lineDirection,lineDirection));

Vector3 direction;

direction.x = a.x + (lineDirection.x *t) - point.x;

direction.y = a.y + (lineDirection.y *t) - point.y;

direction.z = a.z + (lineDirection.z *t) - point.z;

float ShortestDistance = sqrtf((direction.x*direction.x)+(direction.y*direc tion.y)+(direction.z*direction.z));

return ShortestDistance;

}

sorry for long post but I will appreciate your patience to read and helps and insights. where I did wrong.