gluUnproject return values depend on cam rotation

Hello.
Im working on a map editor.The map is basically made of cubes of the same size. I also need determine on which side user clicked on cube (i use OpenGL selection buffer) and it seems to work fine.
But i also need to get coordinate(s) of the cube(s) on which user clicked. gluUnproject does that but with a bug.
The returned coordinates depend on camera location / rotation.

Look the pic, the bomb texture should be always on top of the cube, no matter which angle camera (gluLookAt) has and no matter what side of cube user clicked.
http://img836.imageshack.us/img836/4720/badcoord.jpg

You see there is a red cube (the arrow points to it). Like you see i clicked on the bottom of the cube and bomb texture appeared away from the cube. The Y coordinate was increased by 1.

Now look at the second pic:
http://img694.imageshack.us/img694/3451/goodcoord.jpg

Here i clicked on right side of the cube and bomb texture appeared in the right place.
Clicking on TOP of the cube (ie. LID) seems to work always fine. I get XYZ without any “+1” added to them.

But whenever i rotate the cam (cam = gluLookAt) the weird things will happen.
I mean now clicking on right side causes adding by 1 and bottom side works fine. Same with left side and vice versa.

For example look at this pic and pay attention to camera rotation:
http://img94.imageshack.us/img94/8452/badc21.jpg
The bomb texture again is in wrong Y.

Second pic with camera moved “down” and cube is now little bit more above the camera.
http://img38.imageshack.us/img38/3574/goodc21.jpg

Cube LID is textured correctly, bomb texture is in right place.

Can somebody please tell me what causes this “OFFSET” bug that depends on cam location?

Any ideas at least what could be wrong?
Maybe it has something to do with matrices?

I appreciate any help with that bug.
TIA.

Can you post your picking code because the issue lies there.

Unfortunately picking is using pretty “standard” code.
Particulary my code:

Vector3D GetOGLPos(int x, int y){
	GLint viewport[4];
	GLdouble modelview[16];
	GLdouble projection[16];
	GLfloat winX, winY, winZ;
	GLdouble posX, posY, posZ;
	
	glGetDoublev( GL_MODELVIEW_MATRIX, modelview );
	glGetDoublev( GL_PROJECTION_MATRIX, projection );
	glGetIntegerv( GL_VIEWPORT, viewport );
	
	winX = (float)x;
	winY = (float)viewport[3] - (float)y;
	glReadPixels( x, int(winY), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ );
	
	gluUnProject( winX, winY, winZ, modelview, projection, viewport, &posX, &posY, &posZ);
	
	Vector3D v;
	v.x = (double)posX;
	v.y = (double)posY;
        v.z = (double)posZ;
	
	return v;
}

Sometimes when i mess around with different data types:
GLfloat, GLdouble etc, the coords are changing but the bug remains.

It’s very frustrating. Whatever i try with picking, every method just fails or has bugs in it.
I don’t think that this is bug in OpenGL but a bug in code or a bug in OpenGL initialization code. glDepthFunc etc.

Im not so experienced with OpenGL yet to figure out what causes this.

That’s why i ask you experts here.
If you have some ideas about what can cause such anomalies, or anything in mind.

Please post it.

I looks like you have a problem when you are rounding the 3D coordinates to get the integer position of your cube.

What do you mean by that.
Rounding problem in GetOGLPos?

I have seen so many different sources. So that i don’t even know which is the right way to return the correct values.

I indeed need INTeger values of cube coords.
Im doing a glTranslatef with those INT coordinates.

Can such a small thing really cause that problem?
Why it depends on camera location (rotation) then?

How i should check my Z values from glReadPixel?
If i click on background of my scene, where nothing is drawn.
Then the Z is 1.

If i click anywhere else on my scene where there is something rendered, i get values like 0.237642343, 1.4823478623

etc etc.

Is this normal?

No your GetOGLPos looks fine.

The Z you get is a function of the distance in the view (dependent on rotation and translation) between 0 and 1. If you get something higher than 1 then your Z-buffer is corrupt. Where nothing is drawn, you are at maximum Z, which is 1.0.

gluUnProject is dependent on the camera position and orientation because your rendering is.

What I mean by rounding error, is in the code following your picking function.
If you click on the front face and get a value of say 1.0 +/- epsilon and then click on the back face you might get 2.0 +/- epsilon. If you round incorrectly to get your cube position, you will get one next to it.

If you need more accurate selection I suggest you do some sort of false color rendering, where each selectable object is rendered with one different color, coding its ID. Then gluUnProject is not needed. This rendering does not need to be displayed.

So in my case the 1.4823478623 and such values are invalid for Z ?

You said Z values are in range of 0…1.
So values like 1.something (1.24345) are invalid?
It never shouldn’t be higher than 1.0 but always between 0…1 ?
Correct?

And values like my example above 0.237642343 until 1 are valid.
Im correct?

Can you tell me the correct way? If you know it of course.

I know about color picking but what about coordinates?
Whole editing (map format) in my editor and for the particular game relies on coordinates. If i don’t know the coordinate of the cube the user is editing i can’t edit it.
OK, i can edit it but i don’t know where im editing. So as a result, i can’t save such map.

Well… It’s a bit difficult to explain but without coords i can’t do much.

Btw: what can cause of Z buffer corruption?
Never heard of it. But it sounds weird.

I know i have alot to learn about 3D, OpenGL, Z buffer etc.

But im thankful that at least somebody tries to help with my problem.

Thanks.

Yes values in the Z-Buffer should stay between 0 and 1.

If you look at the OpenGL spec you can see the formula at the section about ‘Coordinate Transformations’ and ‘Controlling the Viewport’.

For the coordinates of the cubes, you can store for each cube its coordinates in a std::vector for example, then use the cube index in the vector as color for the false color rendering. When you get the color for selection you in fact have the index of the cube, and the coordinates are stored in the vector.
If you need the exact coordinates (precision higher than cube position) then you indeed need to use gluUnProject.

Maybe a combination of false color and gluUnProject can disambiguate where you are clicking.

For the correct rounding method, that depends on how your cubes are arranged, like for example between integer coordinates (between i and i+1), centered on integer coordinates (between i-0.5 and i+0.5) or any other convention you use.

Maybe if you could post the “small” part that gets you the cube coordinates from your gluUnProject 3D coordinates I could tell you.

Hope this helps.

Cubes are drawn like this:

for (int x = 0; x <= 255; x++) {
        for (int y= 0; y <= 255; y++) {   
          glPushMatrix();
          glTranslatef(1.0f*y, 0.0f, 1.0f*x);  
        
          for (int z=0; z < 5; ++z) {            
            glPushMatrix();              
              drawBlock(getTheBlockAt(y, x, z));
            glPopMatrix();

            glTranslatef(0.0f, 1.0f, 0.0f);
          }
          glPopMatrix();
        }
      }

Maybe if you could post the “small” part that gets you the cube coordinates from your gluUnProject 3D coordinates I could tell you.

I just typecast the values from GetOGLPos to integer.
For ex:

Vector3D vv; 
vv = GetOGLPos(mouseX, mouseY);
int X,Y,Z;

X = (int)vv[0];
Y = (int)vv[1];
Z = (int)vv[2];

This is very bad way i guess. As i said it works somehow but it’s not precise.

As you see im doing a:

glTranslatef(1.0f*j, 0.0f, 1.0f*i);

So what can we do about it?

Thank you.

I just typecast the values from GetOGLPos to integer.

That’s where your problem seems to be.

With gluUnProject you will get sometimes 1.999 and other times 2.001 instead of 2.
In the first case, your typecast will get you 1 instead of 2.

You can try: X = (int) (vv[0] + 0.5); and so on.
or X = rint(vv[0]) if you have that function available.

Thanks, with 0.5 it was just too “away” from actual click point, but with +0.2 it works pretty fine for 2 sides (left and upper).

If you look at the cube almost completely from top (ie where the LID is).
But for RIGHT and BOTTOM face it still has the bug.
Strange.

What do you think about this: i have one strange thing, the GetOGLPos must have it’s Y and Z swapped. I mean Y is returned in Z and Z is returned in Y.

The return variables in gluUnproject are in correct order. But for me they must be swapped for the Y, Z values to be correct.
Do i have a bug in my code somewhere?

I think maybe this also messes with the bug im working on now (clicking on cube sides).

The sourcecode for editor is pretty big and complicated, but i think im drawing whole scene rotated and upside down.
It renders correctly but something is bad in there.

I have a glRotatef(180, 1, 0, 0); at the beginning of my rendering func.

Edit: i have made another test app with cubes and adding +0.2 to the coords and in test app clicking on any sides works. Doesn’t matter what the rotation of cam is.

It has some slight “offsets” when clicking (ie sometimes it jumps to wrong cube) but in general it’s much better.

I had a plan to rewrite the editor anyway so i will make it more “clear”.

Now if you know how to make it even more precise, then tell me also.

There is probably some math needed but i don’t even know which keywords to use in G00gle.

Anyway i will mess around with the numbers and will see how it comes out.

Thanks.

EDIT2 One idea i had in mind is to get the region of clicked cube (it size should be 1.0 or 0.5) divide the value by 2 to get center of cube and return that as coord.
When clicking, it will calculate the are of cube clicked and if it’s in area, this cube had it’s click for sure.

Well something like that, pseudo idea.

Just to inform you and others.
I found a excellent sourcecode from one French programming site.
It’s very basic modeling program.
Sourcecode isn’t very big but it has the code i needed so much.

The detection of cube sides at any rotation and angle, it uses some nice methods but my 3D math is pretty poor to understand it fully. But it does it’s job very well.

Link:

http://files.codes-sources.com/fichier.aspx?id=41851&f=MonProjet-0.15%5Caide%5CAfAr%EAtes.html

Small code snippet:

GetOGLPos();
if(vueAct==0) { // Are we in Perspective Mode?
      mat44(modelview44,modelview);
      v4[0]=0.0;  // ce point initial en ( 0.0 , 0.0 , 1.0 ) 
      v4[1]=0.0;  // doit źtre mis en ( w4[0] , w4[1] , w4[2] ) 
      v4[2]=1.0;  // pour apparaītre fixe sur l'écran quand on
      v4[3]=1.0;  // change la perspective.    Donc w4  est la
      mulvec44(w4,modelview44,v4); // direction de l'observateur.                        
      oeuil[0]=w4[0]*dist;
      oeuil[1]=w4[1]*dist;  // Current point of the observer oeuil
      oeuil[2]=w4[2]*dist;
      dir[0]=posX-oeuil[0];
      dir[1]=posY-oeuil[1];  // Direction of the target
      dir[2]=posZ-oeuil[2];
      oo = oeuil[0]*oeuil[0]+oeuil[1]*oeuil[1]+oeuil[2]*oeuil[2];
      od = dir[0]*oeuil[0]+dir[1]*oeuil[1]+dir[2]*oeuil[2];
      newX = oeuil[0] - dir[0]*oo/od;  // point aligned with the target point
      newY = oeuil[1] - dir[1]*oo/od;  // in the plane passing through the origin
      newZ = oeuil[2] - dir[2]*oo/od;  // and perpendicular to the direction of view
      if( (sqrt(posX*posX+posY*posY+posZ*posZ) < 2.5) && (option[0] == 1) ) {
         newX = posX;    // in case of perspective and if you have referred 
         newY = posY;    // a point on the surface of the object, then
         newZ = posZ;    // again if the option of choice provides.
         }   

This is not all of course, there is a bit more.
But GetOGLPos() is same as mine.

And fortunately the editor uses very simple mesh format for saving and loading.

Basically just saving coord values of glVertex3f(x,y,z) to file and reading them back.

So the adoption to my editor should be pretty simple.