To a certain extent, the texture proxy mechanism can give you some hints about whether a texture could fit in video mem or not.
You can also assign priorities to textures to help the implementation decide which textures should stay in video mem (note that priorities are just hints, and can be ignored by the implementation). To know if some textures are resident, you can use glAreTextureResident(). That’s about all you can do in OpenGL to control video memory usage.