I wrote some code a while ago to click-test and marquee select 3-d triangles in viewport space; to test the triangles of my 3-d model, I apply the combined mode-view + projection matrix, perspective divide, and then the triangle test is easy and cheap, because it's basically 2-d. The transformed Z coordinate provides ordering to select the nearest triangle. It seemed like a good idea at the time.
Recently I hit an unfortunate fail case: if the triangle being tested goes through the near clip plane and behind the camera, the vertex with positive Z (in eye coordinates) becomes flipped after perspective divide, causing my 2-d triangles tests to fail pretty badly.
My best band-aid idea right now is to clip the triangle against the near clip plane...but I was wondering...
1. Is it possible to do a triangle-ray intersection test entirely in clip space (using homogeneous coordinates) and
2. If the triangle goes behind the camera, will the intersection test produce something useful (e.g. an intersection point that can be clearly identified as fictitious if the unclipped part of the triangle doesn't intersect the test point) and
3. If the intersection actually is useful, are there more optimized hit-tests than intersecting the triangle's supporting plane and then checking the returned point? (For example, is it possible to find the barycentric coordinate of the intersection if the inputs are all in homogeneous coordinates)?
I did a bunch of googling but didn't find anyone doing intersection tests except in cartesian coordinates.