I am writing a C++ matrix library, mainly to learn more about linear algebra and improve my programming skills, even though I know plenty of libraries already exist. I have used one such library and quickly discovered that if you don't entirely understand how it was structured, it can lead to confusion with basic multiplication. Case in point:

Whether you think of matrices as row-major or column-major, a matrix * matrix has a clear mathematical meaning and implementation.

Thus, if you think of a row-major ABC or a column-major CBA, the actual code you write should be in "calculation order" and thus A * B * C in both situations. Does this make sense? If you instead implemented a matrix library to allow a column-major notated order of C * B * A in actual code, you could easily confuse users (including yourself) since the implementation of "*" would need to vary from the standard mathematical meaning.

But this presents additional confusion when you consider the vector * matrix. If we implement all code in "calculation order" as suggested above, then a user would think of column-major CBAv (v is a vector) as coded like v*A*B*C. Therefore your vector* operation would need to be implemented as if the vector was actually on the right, which is the intention from the notation and the intention from the calculation order.

The problem is that it reads like the vector is on the left, which OpenGL also actually allows in the shader, by treating the vector as a row-vector, even though the OpenGL matrix is column-major (this v*M would then have the same effect as multiplying the column vector * the transpose of the matrix).

So in my library, I am opting to not allow a vector* overload to avoid this confusion. The user might wonder, "am I treating v as a row or a column? is on right or on left?" But providing only Mv in code leads to the bizarre order of A * B * C * v to get CBAv (or vABC).

Of course, in most situations, you wouldn't concatenate a matrix multiplication and a vector multiplication in the same expression. But the reality remains that part of the library (matrix * matrix) needs to be in "calculation order" while matrix*vector needs to be in column-major notation order, considering these issues.

I am curious how some of you have implemented a matrix * vector and matrix * matrix in your libraries with regards to calculation order.