I would like to test for the (partial) visibility of a rectangle in model space, when rendered into viewport space. I'm currently using Gl's GL_SELECT mode, but there are speed issues with this.
Problem reduces to: an algorithm to test for overlap between a convex quadrilateral Q (the projection into viewport space) and a rectangle R (the viewport itself).
If any vertex of Q is inside R, there's overlap (trivial).
If any edge of Q crosses an edge of R, there's overlap (Cohen-Sutherland, perhaps?)
Finally, if Q encloses R, there's overlap (can't see any easy way to do this).
Can anyone help?