when rendering a quad with the following settings, i have the problem that the antialiasing not only occurs on the edges of the quad, but visibly on the edges of the triangles which make up the quad.
The hardware always rasterize triangles.
Polygon smoothing is not the same as MSAA/FSAA. it is a kind of older/less correct ways of antialiasing.
Try disable depth testing, but it will bring other problems.