OpenGL clip planes

I usually would not consider my self a ‘beginner’ when it comes to OpenGL… however, I recently tried to use normal OpenGL clip planes and I simply do not understand how they are supposed to work. Specifically what the plane equation needs to look like, depending on how your view and projection looks like etc.

The documentation states

equation is transformed by the inverse of the modelview matrix and stored in the resulting eye coordinates
Now that right there confuses me already. Usually you multiply eye coordinates with the inverse modelview to get their object coordinates. But in this case, we have to construct a plane equation, that is multiplied by the inverse modelview and the resulting plane vector is used against the eye coordinates of the vertices to determine whether they are in or out.

Let’s say I have a Ortho projection for 2D rendering, something like

glOrtho( 0.0, 100, 0.0, 100, -1.0, 1.0 );

and a viewport like this

glViewport( 0, 0, 100, 100 );

Now I want to clip away anything that is below 20 and above 80 on the x-Axis, what plane equations do I have to use?

As far as I have tried a plane equation like this: -1,0,0,80 would work for the latter, but 1,0,0,20 would not work for the former.

The description in reference documentation is perhaps a bit misleading. The red book uses this notation

(A, B, C, D) M^-1 (x, y, z, w)^t >= 0

to describe which points are “in”, where A, B, C, D are the plane equation coefficients and x, y, z, w are vertex coordinates in eye space. So to do the comparison in eye space, the plane’s normal is transformed with the inverse transpose of the modelview matrix (the transpose is missing from the reference doc), which makes sense if you think about it, because the inverse transpose of the modelview is what is always used to transform normals to eye space.

To clip anything with x < 20 you would use the equation 1,0,0,-20, because you want e.g. the point (20, 0, 0) to be in the plane, in other words substituting these coordinates into the plane equation should yield 0.

Since you are working in ortho projection, and you are trying to do screen space clipping, I suggest you use scissor test to clip away unwanted pixels.