Camera orientation issues

Hi all,
I am working to create my own view matrix and what I have is a camera direction (dir) and a side vector on XZ plane (Y being the OpenGL up).

up = side x dir / ||side x dir||
side = dir x up / ||dir x up||

matrix =
( sx, sy, sz,
ux, uy, uz,
-dx, -dy, -dz )

This matrix ends up being inconsistent.
There is a certain side direction for which the camera rotates fine as direction changes. But other than that, for instance if the side direction is the negative of that, camera rotates the opposite of the direction vector. And it between it’s just skewed. How can I make it more consistent?