gluUnproject and persp projection

[mjs1]

Hello all!

I'm trying gluUnproject for the first time to turn mouse coordinates into world space coordinates. I have it working like a charm for orthographic projections but am having trouble getting the world coordinates I expect when using a perspective projection. For example, with the code below, I get world coordinates of -19.0, -19.0, -19.0 when I click on what should be the world center. The good news is that these numbers change in the correct relative sense. For example, if I click on an area above the world center, the y coord will increase in value.

Here's my sample code. I listed the glutMouseFunc() and glutDisplayFunc() functions I am calling. It's set up to produce the perspective projection, but I also have the two lines necessary to produce the correctly-working ortho projection commented out. In order to post this I had to strip the parentheses out of the gluUnProject command. I kept getting errors about posting html. How are other people posting code fragments?

static void myDisplay(void)
{
glClearColor(0.6, 0.6, 0.6, 1.0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

glMatrixMode(GL_PROJECTION); glLoadIdentity();
//gluPerspective(45.0, 1.0, 1.0, 50.0);
glOrtho(-2.0, 2.0, -2.0, 2.0, 0.1, 100);

glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
//gluLookAt(10.0, 10.0, 10.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0);
gluLookAt(0.0, 0.0, 5.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0);

glutWireSphere(1.0, 10, 8);

glutSwapBuffers();
}

static void myMouse(int button, int state, int x, int y)
{
GLint viewport[4];
GLdouble model[16], projection[16];
GLfloat winX, winY, winZ;
GLdouble posX, posY, posZ;

glGetIntegerv(GL_VIEWPORT, viewport);
glGetDoublev(GL_MODELVIEW_MATRIX, model);
glGetDoublev(GL_PROJECTION_MATRIX, projection);

winX = (float)x;
winY = (float)viewport[3] - (float)y;
winZ = 1.0;

cout << "window coords are: " << winX << "\t" << winY << "\t" << winZ << endl;

gluUnProject winX, winY, winZ, model, projection, viewport, &posX, &posY, &posZ;

cout << posX << "\t" << posY << "\t" << posZ << endl;
}

[Dol AI with 3D Bone system]

Do you have the resize code?

In my test case I didn't have a resize func. I just had the mouseFunc, displayFunc, and main. I have a resize in my actual code but taking it in the test case out did not change the results I was getting from gluUnproject.

[Dol AI with 3D Bone system]

Tomorrow I will post my code to exchange idea with you.

[mjs1]

Is the reason this is working in the ortho projection but not the perspective projection due to the value winZ is set to? (1.0). Is this where glReadPixels comes in? Is this why the persp view is having problems?

Is there anything else I need to activate besides GL_DEPTH to have glReadPixels() to work correctly?

[mjs1]

I got it! GL_DEPTH_TEST was not enabled. Working fine now. My only question now is how wireframe or point objects are generally depth tested. It's easy to get a correct depth hit when using shaded objects. When using wireframes or points it's a lot harder. When using gluPickMatrix() you can specify a window in which to search. Do people normally do a few glUnprojects with x, y values close to the input x, y value and take the one with the nearest depth?