Fit orthogonal projection matrix around shadow casters (CSM)

I have already integrated an algorithm to find all possible shadow casters for a directional light, in and around the camera frustum. I then calculate the area that creates the tightest fit around the casters and create the orthogonal projection matrix from that.

This works, as long as the light is pointing straight up or down. If it's pointing in any other direction, the shadows are stretched, and the objects end up being rendered outside of the cascade texture maps. This video should make clear what I mean:

http://youtu.be/ihtMfS2V3jk

At the top you can see the cascades. The only shadow caster in this scene is the tree. Ignore the self-shadowing, that's a problem for another time.

At around 10 seconds in I start changing the direction of the light. You can immediately tell by the cascade maps, that the object is being cut off.

My question is: How to I adjust my projection matrix to fit the entire shadows, if the light is pointing at an angle?

I'm assuming I need to transform either the orthographic matrix, or the points it is created from, with the view matrix, I'm just unsure how.