View Full Version : convert 2D mouse coordinates to 3D ray for picking

04-23-2003, 03:18 PM
I am working on developing a 3D rendering engine base both DirectX8.0 and OpenGL,which allow user to select between the two 3D API. I am programing on Win2K+VC++6.0 environment.Now I want to implement ray picking objects by mouse and the objects would be transformed in world space.I known there is a sample about Ray Picking with Object Transformations base DirectX8.0,following as: http://www.mvps.org/directx/articles/improved_ray_picking.htm
code section:
void CMyD3DApplication::Pick()
D3DXVECTOR3 vPickRayDir;
D3DXVECTOR3 vPickRayOrig;

// Get the pick ray from the mouse position
if( GetCapture() )
POINT ptCursor;
GetCursorPos( &ptCursor );
ScreenToClient( m_hWnd, &ptCursor );

// Compute the vector of the pick ray in screen space
v.x = ( ( ( 2.0f * ptCursor.x ) / m_d3dsdBackBuffer.Width ) - 1 ) / m_matProj._11;
v.y = -( ( ( 2.0f * ptCursor.y ) / m_d3dsdBackBuffer.Height ) - 1 ) / m_matProj._22;
v.z = 1.0f;

// Get the inverse view matrix
D3DXMatrixInverse( &m, NULL, &m_matView );

// Transform the screen space pick ray into 3D space
vPickRayDir.x = v.x*m._11 + v.y*m._21 + v.z*m._31;
vPickRayDir.y = v.x*m._12 + v.y*m._22 + v.z*m._32;
vPickRayDir.z = v.x*m._13 + v.y*m._23 + v.z*m._33;
vPickRayOrig.x = m._41;
vPickRayOrig.y = m._42;
vPickRayOrig.z = m._43;

// calc origin as intersection with near frustum


// iterate through the scne and find closest match

CObj *hitObj=m_pScene->Pick(&vPickRayOrig,&vPickRayDir,FAR_Z-NEAR_Z);
CObj *CObj::Pick(D3DXVECTOR3 *pvNear, D3DXVECTOR3 *pvDir, float maxDist, CObj *pObj)
// clear the hit flag

// do we have a mesh?
if (m_pMesh) {

// yes, convert ray to model space
D3DXVECTOR3 vNear,vDir;

// test for intersection
BOOL bHit;
DWORD dwIndex;
float u,v;
float dist;

// is there an intersection?
if (bHit) {

// yes, is it beyond the far plane?
if (dist<maxDist) {

// no, have we previously found a hit?
if (pObj) {

// yes, but is this one closer?
if (pObj->m_fHitDist>dist) {

// yes, clear hit flag of object with previously detected intersection

// set our hit flag

// store distance to the hit

// set hit object pointer to point to this object

// no, this is the first hit
} else {

// set the hit flag

// save distance to intersection

// set hit object pointer to point to this object

// perform pick operation on children and siblings
if (m_pChild)
if (m_pSibling)

// return hit object pointer to caller
return pObj;
But I don't known how I can implement same function base OpenGL because there is no independent world and view transformation matrix can be get.Can anybody give me some hints about this?The key problem I want to solve is how to convert 2D mouse to a 3D ray.
Then I can test intersection between the objects of scene and the ray and determine the selected objects.

Thanks in advance
Best Regards

04-23-2003, 04:47 PM
This is a beginner question.

You can get the PROJECTION and MODELVIEW matrices with glGetDoublev. You can get the VIEWPORT with glGetIntegerv.

Then you can use gluUnProject twice, once with z=0 and once with z=1 to get the possible world coordinates of your 2D point at the near and far planes.

Those two points define the ray you're looking for.

[This message has been edited by OneSadCookie (edited 04-23-2003).]

04-23-2003, 07:14 PM
Firstly, I agree with OneSadCookie...

Secondly, look up select buffers, they may do what you want to do.

Thirdly, think of your viewport (window/screen...) as your near plane and your "camera" as the point at which your ray will start. Your near plane is a quad that is defined by you and you should be able to determine it's coords/dimensions.

You also know the resolution of the screen (say 800x600). Given this, you should be able to work out the end point of the ray on the near plane (I've got some code that does this somewhere but can't be knotted looking for it just now). Once you've determined this you just extend the point out past the limits of your scene or the far plane (whatever suits).

This gives you a ray in view-space which you can then compare to the various bounding spheres/objects/poly's/vertices etc. in your scene (once you have transformed the verts into view space) and gives you any level of detail with respect to picking that you care to code.

I've used this method to allow the placement of "joints" onto the surface of objects - ie. so various objects can be joined together using these joints.