There are two alternatives for rotational transformations: quaternions or matrices. I like matrices. But multiplying them many times produces numerical error which makes their vectors flow away from unit length and orthogonality. Therefore orthonormalization has to be performed eventually after n transformations.

The easy way is this:

mat3 RotMtx;

vec3 i = RotMtx[0];

vec3 j = RotMtx[1];

vec3 k = RotMtx[2];

vec3 Ox = normalize(i);

vec3 Oz = normalize(cross(i,j));

vec3 Oy = cross(Oz,Ox);

RotMtx[0] = Ox;

RotMtx[1] = Oy;

RotMtx[2] = Oz;

But this is not the ideal solution, because one axis' direction is preserved (i), another changed (k->Oz), and the last one (Oy) does not really depend on it's original direction at all - it may even form a right-hand vector triplet out of left-hand one.

The perfect solution, as I think, must be equal to all three input vectors: the direction of each one must change by the same angle to form an orthogonal triplet, and it shouldn't be an iterative algorithm. But I just can't find a solution!

So simply saying, the task is the following: we have three non-collinear vectors, a,b,c (let's say, they are of unit length); we need to make them orthogonal by rotating each of them by the equal angle.

Here is what we have:

|a|=|b|=|c|=1;

a,b,c are not collinear

a -> A

b -> B

c -> C

Here is what we need:

dot(A,B)=0; (1)

dot(A,C)=0; (2)

dot(B,C)=0; (3)

|A|^{2}=1; (4)

|B|^{2}=1; (5)

|C|^{2}=1; (6)

dot(a,A)=dot(b,B); (7)

dot(b,B)=dot(c,C); (8)

As you can see, I came up with 8 equations, but there are three unknown 3-dimensional vectors, which means 3*3=9 equations required. What do I miss?

And maybe there are some geometrical solution exists? Any tips, people?..