Video Memory Fragmentation

One thing passed through my mind today while coding a texture cache for my 3D engine :
How is the video memory fragmentation managed ?
I mean, when you send several different sized textures to the graphic card, then replace them by others, memory must be fragmented after a while.
How can you control this, and what is the remedy ?