Mouse coordinates into World coordinates

[fiodis]

I'm trying to draw a quad on a 3D terrain under the mouse. To do this, I need to convert the mouse's window coordinates to world coordinates somehow. I Google'd the subject and found a bunch of tutorials, but they all talk about gluUnProject and legacy code. I'm using OpenGL 3.1. Is there any way to convert mouse coordinates to world coordinates without using legacy code?

[tonyo_au]

Have a look at glm unproject

[rogxx]

// This give You a line normal to screen, passing through the mouse pointer, in modelview coords.
// myln1 declared as: double myln1[8];
// pt_tmp as: double pt_tmp[4];
// mymouse[] are X and Y coords of mouse pointer in pixel (0,0 top - left).
// oglW and oglH are opengl window width and height in pixel.
 
  myln1[0] = (double)mymouse[0] - ((double)oglW/2.0);
  myln1[1] = (double)mymouse[1] - ((double)oglH/2.0);
 
  myln1[0] = myln1[0] / scale_factor;
  myln1[1] = myln1[1] / scale_factor;
  myln1[0] = myln1[0] / ((double)oglH/2.0);
  myln1[1] = -(myln1[1] / ((double)oglH/2.0)); // I use a PRJ axis right-handed,
                                               // opengl has the PRJ left-handed
 
  myln1[2] = -100.0; // Chose Z from screen as You need.
 
  myln1[4] = myln1[0];
  myln1[5] = myln1[1];
  myln1[6] =  100.0; // Chose Z into screen as You need.
 
  pt_tmp[0] = myln1[0] - MODELVIEWmatrix[12];
  pt_tmp[1] = myln1[1] - MODELVIEWmatrix[13];
  pt_tmp[2] = myln1[2] - MODELVIEWmatrix[14];
  Vec3XtraspMatr33(pt_tmp, MODELVIEWmatrix, myln1);
 
  pt_tmp[0] = myln1[4] - MODELVIEWmatrix[12];
  pt_tmp[1] = myln1[5] - MODELVIEWmatrix[13];
  pt_tmp[2] = myln1[6] - MODELVIEWmatrix[14];
  Vec3XtraspMatr33(pt_tmp, MODELVIEWmatrix, &myln1[4]);
 
// Now You have start and end points of a line normal to screen,
// passing through the mouse pointer, in modelview coords.
// Start point x,y,z: myln1[0], [1], [2]
// End point x,y,z:   myln1[4], [5], [6]
 
// *************************************************************************
// Here is the subroutine Vec3XtraspMatr33():
// --- Vector (or xyz point) * transposed matrix: ---
// par 1: InpVal, vect [0],[1],[2]
// par 2: InpVal, matr [0],[4],[8]
//                     [1],[5],[9]
//                     [2],[6],[10]
// par 3: RetVal, vect [0],[1],[2]
void Vec3XtraspMatr33(double *vec_1, double *mat_1, double *vec_r)
  {
  vec_r[0]=vec_1[0] * mat_1[0] + vec_1[1] * mat_1[1] + vec_1[2] * mat_1[2] ;
  vec_r[1]=vec_1[0] * mat_1[4] + vec_1[1] * mat_1[5] + vec_1[2] * mat_1[6] ;
  vec_r[2]=vec_1[0] * mat_1[8] + vec_1[1] * mat_1[9] + vec_1[2] * mat_1[10];
  }

I hope this can help You

[rogxx]

Sorry, I forgot to say that this work with a PRJ matrix created by glOrtho(), and I use it only for zooming, I never transl or rotate the Prj matrix.

[fiodis]

Thanks for the code sample, but I've since figured out how to use glmUnProject and it's giving me the right results. On thing that tripped me up initially, though, was that even when you're rendering solid triangles, if you set the polygon mode to LINES, you only get the right results if you click on a line, and not the empty space in the middle of a polygon.

Regarding a line normal to the screen - wouldn't that only work in 2D space?

[rogxx]

Regarding a line normal to the screen - wouldn't that only work in 2D space?

No, You get a real 3D line in MODELVIEW matrix coords, and, with rotation and translation of MODELVIEW, at the time You click the mouse it is normal to the screen.
For selection (picking) I use the pre-built opengl utility, with display lists, but if I select a segment, and I want to know which end of the segment is closer to the point(line) of the click, I use the code above, then I find the intersection point(or near-intersection points) between the segment and the mouse generated line.