How to anti-alias images shown as 2D textures?

Igor_Levicki · January 7, 2010, 9:00am

I am writing an image viewer application. It is pretty simple, and it uses OpenGL to display photos (as a 2D texture), enabling very fast zoom/pan without tearing.

I wanted to improve zoom quality (particulary zoom out or downsampling) so I enabled multisample. With 8x MSAA (which is maximum I can get for some reason on ATI HD 5850 with Catalyst 9.12) there is some improvement, but it is very subtle.

What would be a recommended way of letting OpenGL downsample the 2D texture containing the image with highest quality?

Would super-sampling help? If so, how to enable it?

If not, what else apart from writing a shader or downsampling on a CPU?

mfort · January 7, 2010, 9:36am

Look for Mipmaping. Antialiasing does not help in downsampling.

Igor_Levicki · January 7, 2010, 3:27pm

Funyy, by reading this:

http://www.opengl.org/registry/specs/ARB/multisample.txt

Especially the part:

This extension provides a mechanism to antialias all GL primitives: points, lines, polygons, bitmaps, and images.

I was under impression that it would work on textures as well being that the textures are images.

Sure, I can Google how to setup mipmapping, but how to select the best filtering method for downsampling (and upsampling?), and how to turn off “performance” optimizations that hurt image quality on ATI and NVIDIA cards? Will mipmapping work for NPOT textures?

zed · January 7, 2010, 4:14pm

>>Would super-sampling help? If so, how to enable it?

you need to draw into a texture larger than the window eg 4x larger (use FBO or something)

and then draw that texture to window

as mfort saiz MSAA only really helps on polygon edges, for photos youre not gonna see much benefit

look into mipmapping and perhaps also negative mipmap bias
http://www.opengl.org/registry/specs/EXT/texture_lod_bias.txt

Igor_Levicki · January 7, 2010, 4:18pm

I am trying to keep it as simple as possible.

From what I just read on gluBuild2DMipmaps() it does not work with non-power of two textures (it scales the original to the nearest power of two before generating mipmaps) so that doesn’t work for me.

Any other way to generate NPOT mipmaps in OpenGL?

Generating them on CPU would slow down the time it takes to display the photo initially.

How to use cubic (or better) filters instead of box?

Zeno · January 7, 2010, 4:34pm

Don’t use glu to make mipmaps. If you’re not pre-calculating them use this:

glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP_SGIS, GL_TRUE);

Spec here: http://www.opengl.org/registry/specs/SGIS/generate_mipmap.txt

Cubic filters can be programmed in a shader. There’s an article on it here: http://http.developer.nvidia.com/GPUGems2/gpugems2_chapter20.html

Dark_Photon · January 7, 2010, 7:11pm

Super-sampling (SSAA) might help. Multisampling (MSAA) would not.

Besides the technique zed mentions, some vendors offer SSAA GL modes you can draw to. This would be a quick way to test for gain without having to code up FBO render and downsample.

For instance, on NVidia GTX285 on Linux with latest drivers:

> nvidia-settings --query=fsaa --verbose
...
   Valid 'FSAA' Values
      value - description
        0   -   Off
        1   -   2x (2xMS)
        5   -   4x (4xMS)
        7   -   8x (4xMS, 4xCS)
        8   -   16x (4xMS, 12xCS)
        9   -   8x (4xSS, 2xMS)
       10   -   8x (8xMS)
       12   -   16x (8xMS, 8xCS)

So 9 is supersampling. You can nail this on with nvidia-settings or setting the _GL_FSAA_MODE env var to 9 before running the app.

But try zed’s other suggestions.

Also, are you always displaying these images with the texel axes aligned to the pixel axes (that is, no rotating of images clockwise or counterclockwise on the screen, and no “tipping” images into or out of the screen?) If not, crank up your anisotropic filtering.

Igor_Levicki · January 8, 2010, 8:20am

Mipmaps require some coding so it will take me some time to try that.

No, I am not rotating or tipping the image, or otherwise I would need anisotropic filtering as well.

Regarding super-sampling, that was exactly my thought – quick and dirty solution to the problem.

However, I am at a loss how to enable it on ATI HD 5850?

Their OpenGL documentation is dated 2005 so no help there.
If I enable multisample, the highest value for WGL_SAMPLES_ARB I can use is 8 or wglChoosePixelFormatARB() returns zero formats, and 8 coincides with 8x MSAA.
In Catalyst Control Center there are the following options:

Antialiasing level: can be set to 2x, 4x, 8x
Filter: can be set to box, narrow tent, wide tent, edge-detect

When you set 8x and box it says 8 samples.
When you set 8x and narrow tent it says 12 samples.
When you set 8x and wide tent it says 16 samples.
When you set 8x and edge-detect it says 24 samples.

So, I presume that setting WGL_SAMPLES_ARB to 8 and somehow selecting filter to be edge-detect might enable supersampling but I don’t have a clue if that is true and if so, how to do it?

EDIT:
That feature is something ATI calls Custom Filter AA (CFAA).
Basically, you enable it from Catalyst Control Center and then it works for all applications. Now I have additional questions:

Is there a way to enable edge-detect filter programmatically from OpenGL?
Since there is the possibility for using custom AA filters available, I would like to know how do I supply my custom filter in OpenGL?

Alfonse_Reinheart · January 8, 2010, 1:22pm

You could generate mipmaps for the texture by calling glGenerateMipmaps(EXT if you’re using EXT_FBO) on the texture.

ZbuffeR · January 8, 2010, 1:25pm

As said Zeno above:
glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP_SGIS, GL_TRUE);
Or alphonse :
glGenerateMipmaps();

poof auto mipmaps. Not that hard, really.

Igor_Levicki · January 8, 2010, 4:29pm

Do those mipmaps work for non-power-of-two textures?
What downsampling algorithm is used to generate them?
Can I change the algorithm to get the better quality?

Alfonse_Reinheart · January 8, 2010, 4:32pm

Do those mipmaps work for non-power-of-two textures?

Unless you’re using old ATI hardware, yes. All NPOT does is remove the power-of-two restriction.

What downsampling algorithm is used to generate them?

Whatever the implementation wants. It’s a convenience function.

Can I change the algorithm to get the better quality?

No. You can implement it yourself if you want.

Igor_Levicki · January 11, 2010, 4:22pm

What do you mean by “implementation”?

You mean like in “replace the algorithm that mipmap generating function uses by calling some other obscure OpenGL function and passing a function pointer” or like in “implement it on a CPU without using OpenGL function at all”?

Alfonse_Reinheart · January 11, 2010, 4:31pm

What do you mean by “implementation”?

The OpenGL implementation.

OpenGL is just a specification, a document. It is implemented by NVIDIA, ATI, Apple, and others. Your code runs on a particular implementation of OpenGL usually written by the makers of your GPU.

The OpenGL specification allows certain features to be “implementation-defined”. This means that the results are allowed to vary from implementation to implementation, though usually within certain expectations.

You mean like in “replace the algorithm that mipmap generating function uses by calling some other obscure OpenGL function and passing a function pointer” or like in “implement it on a CPU without using OpenGL function at all”?

Actually I meant, “Write a shader to scale the image down and render this to the lower mip level of the texture.” You could do it on the CPU if you want, but I don’t really see the need to.

Igor_Levicki · January 14, 2010, 3:31pm

Ok thanks everyone for the help.

Seeing how I want to control quality of downsampling and OpenGL doesn’t provide such control, and given that it is hard to make GLSL work on ATI I will have to reconsider whole approach and perhaps reimplement this viewer in DirectX to see if it is going to be easier make it work the way I want.

ZbuffeR · January 14, 2010, 4:10pm

Huh, I am curious to learn about how “control quality of downsampling” can be easier on Direct3D …

Alfonse_Reinheart · January 14, 2010, 5:20pm

given that it is hard to make GLSL work on ATI

You’re doing downsampling. It’s texture accessing and some basic vector math. That stuff works even on ATI drivers.

Huh, I am curious to learn about how “control quality of downsampling” can be easier on Direct3D

Does D3D even provide automatic mipmap generation?

Igor_Levicki · January 15, 2010, 12:43pm

I certainly hope so.

Does this answer your question?

Igor_Levicki · February 23, 2010, 12:33pm

I have to bring this topic back from the dead:

People suggested me to use:


glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP_SGIS, GL_TRUE);

But the problem is that I am using GL_TEXTURE_RECTANGLE_ARB because I need my textures to be non-power-of-two.

Furthermore, this is not enough to implement mipmapping, one has to call glGenerateMipmap(), but that function:

Doesn’t work with GL_TEXTURE_RECTANGLE_ARB.
Requires that width and height are powers of two.

Now, either I have some fundamental misunderstanding of all this, or the OpenGL API has a serious problem, and I somehow believe it’s the later.

What good are non-power-of-two textures if you can’t use them throughout the API? Who in their right mind decided that those do not need filtering?

ZbuffeR · February 23, 2010, 12:58pm

non-power-of-two textures is very different from texture rectangle.
http://www.opengl.org/wiki/NPOT_Textures
Rectangle can’t have mipmaps, NPOT can.

Furthermore, this is not enough to implement mipmapping

Wrong, only one is needed. GL_GENERATE_MIPMAP_SGIS and glGenerateMipmap are different : the first tells GL to generate mipmaps whenever texture data changes and is deprecated in GL 3+ ; the second triggers mipmap generation once, at the moment it is called, and is the preferred modern approach (useful when doing multiple passes of rendering to a texture, only generate mipmaps when needed, instead of doing it at each pass)