gluUnProject?

[nhartung]

Right now i'm working on a 2D tile based map editor and i'm attempting to convert mouse coordinates to vertex coordinates (if that makes sense). Basically I want to know which tile the user is clicking on based on the mouse coordinates.

I have been looking for awhile and i think glUnProject is the way to go. Although I could have completely misinterpreted the use of this function. I have looked at a few tutorials and have managed the following attempt using one of NeHe's tutorials:



GLint viewport[4];
GLdouble modelview[16];
GLdouble projection[16];
GLfloat winX, winY, winZ;
GLdouble posX, posY, posZ;

glGetDoublev( GL_MODELVIEW_MATRIX, modelview );
glGetDoublev( GL_PROJECTION_MATRIX, projection );
glGetIntegerv( GL_VIEWPORT, viewport );

cout << mouse_x << " " << mouse_y << '\n';

winX = (float)mouse_x;
winY = (float)viewport[3] - (float)mouse_y;
glReadPixels( mouse_x, int(winY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &amp;winZ );
gluUnProject( winX, winY, winZ, modelview, projection, viewport, &amp;posX, &amp;posY, &amp;posZ);
cout << winZ << '\n';
cout << posX << " " << posY << " " << posZ << '\n';


This is giving me some odd results. The final values are something like -9.25596e+061 for every value. Since this is a just a 2d application I tried using winZ to be zero (it is actually evaluating to 0 anyway when I use glReadPixels) so i'm not sure if this is causing any problems.

[strattonbrazil]

Do your depth values you're reading back seem reasonable? What GUI toolkit are you using? Do you need to flip your y-coordinate to match OpenGL?

[nhartung]

By depth values do you mean winZ? If you do it is returning 0 every time. But considering my game is a 2D tile based game and I'm not using any 3d objects this may make sense. I have an initial offset right now of -5.0 in the Z direction using glTranslatef so i'm not sure whether or not that could effect it. As for a GUI toolkit I don't think i'm using one yet I haven't began working on any GUI. The only outside code I have used is the IJP's library for loading jpg files into textures. (http://www.ijg.org/). And yes I need to flip my y-coordinate to match opengl. Any idea's?

[strattonbrazil]

By toolkit I mean GLUT or GLUI or SDL. The reason I ask is often the toolkit uses a coordinate system where the origin is at the top while OpenGL uses a coordinate system from the bottom, so you need to flip the mouse value.

Can you post what is in your matrices and how they are setup?

[nhartung]

I'm using GLUT which is why I need to flip the y coordinate.

I'm still a complete noob with opengl and I don't understand the matrices very much. I've just been following some tutorials and trying to implement them into my map editor so as far as how I set them up I'm not sure how to explain that. I know I use a vertex array and a texture coordinate array to draw my quads and textures. I'm not sure if this is what you wanted by posting what is in your matrices but I just displayed the values of the array to my program and noticed that they are the same regardless of where I click and the viewport changes when I change my window size, which seems correct. Here is the output for my matrices:

Modelview:
0.25, 0, 0, 0
0, 0.25, 0, 0,
0, 0, 0, 0,
-5.5, -2.875, -5, 1

Projection:
1.29232, 0, 0, 0,
0, 1.73205, 0, 0,
0, 0, -1.0202, -1,
0, 0, -2.0202, 0

Sorry If this isn't what you wanted.

[strattonbrazil]

Those matrices seem fine. I meant more the actual code to set them up. When you say the arrays don't change depending on where you click, your modelview and projection matrix shouldn't be changing.

If you're keeping this in just 2D, it might be more useful just to figure out where you're drawing stuff and check your own boundaries yourself. For example, for a given tile you should know where it sits on your screen and just see if the mouse falls in that region when it gets clicked.

[nhartung]

Yeah, i've been thinking about doing it that way for awhile and after reading this I decided that it probably makes more sense. I just implemented it and it wasn't difficult at all, although it only works for a fixed window size right now but that should work for now. Thanks for all of your help.