model axis and camera orientation

I’m using OpenCV to estimate 3D position of a set of 2D points, with function solvePnP. It’s working for most of the cases, but sometimes the result is flipped (Y axis goes down for the model).

The solution would be a rotation of 180° around X axis for some cases, so I said to myself “OK ! Let’s do some fun dot product between a (0, 1, 0) vector and up camera vector.” But after a little bit of reflexion, both are (0, 1, 0) vectors… But not in the same basis.

So how can I know if my two vectors are not in the same direction ? Both depends on the modelview matrix ? I’m new to OpenGL and these modelview, projection, etc. still a little dark to me.