Does any one know if there is a way to tell the window (device) position [x, y] of a vertex after model-view transformation on that vertex. I want to test if that vertex on screen is within a selecting rectangle. Any help is greatly appreciated.
Well the screen (x,y) coords are not generated right after the modelview transform. After the modelview transform the coords are in eye space (or eye coordinates if you will). Then the clip coordinates are formed after the eye space coordinates are multiplied by the projection matrix. Then the perspective division takes place to generate the normalized device coordinates. Then those are transformed into the window (x,y) coordinates by the viewport transformation which the viewport transformation also happens to be a matrix.
Whew, now that that is done. What you want is to use the selection buffer. This is explained in detail in the Redbook and superbible. Older versions are on the web for free.
-SirKnight
isnt
gluProject() / gluUnproject() (or something like that)
supposed to do that?
If this is about a single vertex and the selection rectangle is small you could try the following:
1)Mask off color/alpha/depth writes, disable depth test, enable stencil writes
2)clear stencil buffer to 0 (or subregion by rendering a quad with the proper stencil ops)
3)set stencil op so to never fail, set 1on write
4)render the vertex as a point
5)readback the selection rectangle region from the stencil buffer
6)check for value 1
This is a fairly ghetto method and will not perform too well.
Use gluProject – very simple and fast and works.
Jeff
Thanks, Jeff. Your method works well.