When you say "downloading", are you talking about copying the data to a GPU buffer (e.g. PBO) or to client memory?

If you're copying to client memory, you need to use glClientWaitSync() to block...