Let P = (Px, Py, Pz, Pw) be the eye-space plane to which you would like to clip.
This will replace your ordinary near plane and should be facing away from the
camera, so Pz < 0.
Let Q be an eye-space point. If P dot Q = 0, then Q lies on the plane P.
What we want to do is modify the projection matrix M so that points Q for
which P dot Q = 0 get transformed in such a way that the transformed z-coordinate
is the negation of the transformed w-coordinate. This corresponds to
the projected z-coordinate that you would ordinarily get for a point lying
on the near plane. After division by the w-coordinate, you get -1.
The projection matrix M transforms points from eye space into homogeneous clip
space. To obtain the clip-space plane P', we need to multiply P by the inverse
transpose of M. (This is because planes are covariant vectors.) The clip-space
plane P' is given by
-1 T
P' = (M ) P
Let Q' be the projection of the point Q, given by Q' = MQ. If Q lies on the plane P,
then P' dot Q' = 0, but we want it to be -Q'w so that it corresponds to the near plane.
So we force it by subtracting 1 from the w-coordinate of P'. Before doing this,
however, we want to divide P' by its z-coordinate as to avoid scaling the z
direction in clip space. Doing all of this gives us the following plane P":
P'x P'y P'w
P" = (-----, -----, 1, ----- - 1)
P'z P'z P'z
We integrate this into the projection matrix M by constructing a new projection
matrix M' as follows.
[ 1 0 0 0 ]
[ ]
[ 0 1 0 0 ]
M' = [ ] M
[ P"x P"y 1 P"w ]
[ ]
[ 0 0 0 1 ]
The matrices M and M' only differ in the entries of the 3rd row, so we could just
calculate replacements for 3rd-row entries of M as follows:
M'(3,1) = P" dot M_1
M'(3,2) = P" dot M_2
M'(3,3) = P" dot M_3
M'(3,4) = P" dot M_4
where M_i means the i-th column of M and M'(i,j) means the (i,j)-th entry of M'.