Now that I have some time, here’s the basic idea,
if you’re using this as your projection transformation:
glOrtho(0, SCENE_WIDTH, 0, SCENE_HEIGHT, -1, 1);
this grabs the region from (0,0) to (SCENE_WIDTH, SCENE_HEIGHT) and linearly maps everything in there into the canonical view volume,the square stretching from (-1,-1,-1) to (1,1,1) (all objects will have a z coordinate of 0 since you’re using an orthographic projection) your viewport transformation will linearly map that square region into a rectangle.
So we have that the mapping from eye coordinates to the cannonical view volume (which I believe are called normalized device coordinates) is linear, and the mapping from there to window coordinates is linear, therefore the mapping from eye coordinates to window coordinates is also linear.
So the bottom left corner of your window corresponds to (0,0) in eye coordinates, and the upper right corner, corresponds to (SCENE_WIDTH, SCENE_HEIGHT) in eye coordinates, and everything in between is proportional.
So you really just need to find out what fraction of the distance across the screen your cursor is, and then find what fraction of the distance across the width or height of your scene that is.
For x, thats easy:
fraction = MOUSE_X/VIEWPORT_WIDTH;
cursor_eye_x = SCENE_WIDTH*fraction;
for y, you have to be careful because MOUSE_Y measures from the top down, and so you have to subtract MOUSE_Y from VIEWPORT_HEIGHT to get the distance from the bottom of the window.
fraction = (VIEWPORT_HEIGHT - MOUSE_Y)/VIEWPORT_HEIGHT;
cursor_eye_y = SCENE_HEIGHT*fraction;
so (cursor_eye_x, cursor_eye_y) gives you where your mouse is currently hovering over in eye coordinates.
I’m going to take a few minutes to test what i’ve written above to make sure my reasoning is correct. But in any case, the important thing is you basically need to keep track of the width and height of both your scene, and of your viewport. I declare them as global variables, and set their values in the ‘reshape’ routine (when you resize the window).
(edit) yes, that seems to be correct. The important thing to remember is that your scene size should generally stay the same, while your window/viewport size can vary. In a 2d setting, I usually set my scene width to 640, and set the scene height so its proportional to the window, in otherwords: so SCENE_HEIGHT/SCENE_WIDTH = VIEWPORT_HEIGHT/VIEWPORT_WIDTH. Solving for SCENE_HEIGHT gives you SCENE_HEIGHT = SCENE_WIDTH*VIEWPORT_HEIGHT/VIEWPORT_WIDTH
all 4 of these values you need to save in order to determine the cursor position in eye coordinates. Thats why I make them global.