I have a transformation matrix for a camera that consists of a Translation and a Rotation. The rotation was a product of yaw and pitch. To make the camera intuitive (at least for me) I decided that vertical translation should be done with Space Bar and C, respectively up and down, no matter where you are looking, so its a translation along the global (0,1,0) axis. As for horizontal translation, A and D count as strafing left and right along the camera's local Left axis, which can be taken from camWorldMatrix[0..2]. As for S and W, they make the camera go back and forth along the normalized projection of the camera's local Z axis onto the XZ plane, which prevents you from using S and W for vertical movement. As for the rotation, pitch is done by rotating the camera around the camera's local X axis, so that you look up and down, but yaw is done by rotating it around the global Y axis, to avoid having roll. I also need to clamp the pitch amount or the local Z axis so that pitch can only go from -90 to 90 degrees instead of doing flips. I've managed to do most of this by only using one matrix for the camera instead of dividing into Translate, Yaw and Pitch matrices or storing an up, side and front vector since they are already stored in the camera's world matrix. The problem is, my mouse controls pitch and yaw using the delta mouse movement, i.e. the cursor's translation between frames, which means I have to grab last frame's camera transformation matrix which is Translation*Rotation and separate them by creating a Translation matrix with camera's position given by camWorldMatrix[0..2] and then multiplying the inverse of that with my Translation*Rotation matrix to get the Rotation matrix. After having the Translation and Rotation separately, I can find the new frame's camWorldMatrix by doing newT*T*newR*R where T and R are the Translation and Rotation matrices I just got and newR is the new frame's Rotation calculated using mouse delta input and newT is the new frame's Translation calculated using keyboard delta input.
Also, for calculating the Rotation matrix out of Yaw and Pitch matrices, I seem to have to do Yaw^-1 times Pitch, otherwise the camera will Roll, I've figured this by doing some algebra but I have no idea why it happens so if someone could explain it to me I would be happy! The Yaw matrix is basically a rotation over vec3(0,1,0) while the Pitch matrix is a rotation over vec3(R*vec4(1,0,0,0)) where R is the previous frame's Rotation.
I'd also like to know if there's a better way to do all this and be able to clamp the pitch between -90 and 90 since I can't do that with my implementation. I think I may be over-complicating :P
EDIT: my framework automatically multiplies all matrices along the tree that leads to the camera and then inverts the matrix to find the View matrix, so I can easily divide the matrix into T and R or T, Pitch and Yaw, if that would simplify things