what is alpha mask,is this in opengl

Can i use it make my rectangle round corner??

If you want to make a rectangle with rounded corners, apply a RGBA8 texture to your quad and use blending.
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

Wherever alpha is 0, it will look totally transparent and wherever it is 255, it will look totally opaque.